Computer Networks and Applications. Application Layer ( , DNS, P2P, Socket Programming)

Transcription

1 Computer Networks and Applications COMP 3331/COMP 9331 Week 3 Application Layer ( , DNS, P2P, Socket Programming) Reading Guide: Chapter 2, Sections

2 Announcements v Lab for Week 3 Socket Programming Practice v Sample Problem Set Did anyone attempt the first problem set? Please do. Solutions soon. Set for application layer is now available v Remember mid-semester exam in Week 6 v First Programming Assignment released this week 2

3 2. Application Layer: outline 2.1 principles of network applications app architectures app requirements 2.2 Web and HTTP 2.3 FTP 2.4 electronic mail SMTP, POP3, IMAP 2.5 DNS 2.6 P2P applications 2.7 socket programming with UDP and TCP 3

4 4

5 Electronic mail Three major components: v user agents v mail servers v simple mail transfer protocol: SMTP User Agent v a.k.a. mail reader v composing, editing, reading mail messages v e.g., Outlook, Thunderbird, iphone mail client v outgoing, incoming messages stored on server mail server SMTP mail server user agent user agent SMTP SMTP user agent outgoing message queue mail server user mailbox user agent user agent user agent 5

6 Electronic mail: mail servers mail servers: v mailbox contains incoming messages for user v message queue of outgoing (to be sent) mail messages v SMTP protocol between mail servers to send messages client: sending mail server server : receiving mail server mail server SMTP mail server user agent user agent SMTP SMTP user agent mail server user agent user agent user agent 6

7 Electronic Mail: SMTP [RFC 2821] v uses TCP to reliably transfer message from client to server, port 25 v direct transfer: sending server to receiving server v three phases of transfer handshaking (greeting) transfer of messages closure v command/response interaction (like HTTP, FTP) commands: ASCII text response: status code and phrase v messages must be in 7-bit ASCII 7

8 Scenario: Alice sends message to Bob 1) Alice uses UA to compose message to 2) Alice s UA sends message to her mail server; message placed in message queue 3) client side of SMTP opens TCP connection with Bob s mail server 4) SMTP client sends Alice s message over the TCP connection 5) Bob s mail server places the message in Bob s mailbox 6) Bob invokes his user agent to read message 1 user agent mail server mail server 6 user agent Alice s mail server 5 Bob s mail server 8

9 Sample SMTP interaction S: 220 hamburger.edu C: HELO crepes.fr S: 250 Hello crepes.fr, pleased to meet you C: MAIL FROM: S: 250 Sender ok C: RCPT TO: S: 250 Recipient ok C: DATA S: 354 Enter mail, end with "." on a line by itself C: Do you like ketchup? C: How about pickles? C:. S: 250 Message accepted for delivery C: QUIT S: 221 hamburger.edu closing connection 9

10 Try SMTP interaction for yourself: v telnet servername 25 v see 220 reply from server v enter HELO, MAIL FROM, RCPT TO, DATA, QUIT commands above lets you send without using client (reader) Implication: one could send forged ed Note: Many SMTP servers will not allow the above interaction without authentication. E.g: try the above with port

11 How to tell a fake ? Examine Long Headers or Raw Source Further reading: 11

12 Securing v STARTTLS: upgrade a plain text connection to TLS/SSL instead of using a separate port for encrypted communication Can be used for SMTP/IMAP/POP v PGP (later in the course) 12

13 SMTP: final words v SMTP uses persistent connections v SMTP requires message (header & body) to be in 7-bit ASCII v SMTP server uses CRLF.CRLF to determine end of message comparison with HTTP: v HTTP: pull v SMTP: push v both have ASCII command/response interaction, status codes v HTTP: each object encapsulated in its own response msg v SMTP: multiple objects sent in multipart msg 13

14 Mail message format SMTP: protocol for exchanging msgs RFC 822: standard for text message format: v header lines, e.g., To: From: Subject: different from SMTP MAIL FROM, RCPT TO: commands! v Body: the message ASCII characters only header body blank line 14

15 Quiz: attachments? v IF SMTP only allows 7-bit ASCII, how do we send pictures/videos/files via ? A: We use a different protocol instead of SMTP B: We encode these objects as 7-bit ASCII C: We re really sending links to the objects, rather than the objects themselves D: We don t!! You have been lied to!! 15

16 Mail access protocols user agent SMTP SMTP sender s mail server receiver s mail server mail access protocol (e.g., POP, IMAP) user agent v SMTP: delivery/storage to receiver s server v mail access protocol: retrieval from server POP: Post Office Protocol [RFC 1939]: authorization, download IMAP: Internet Mail Access Protocol [RFC 1730]: more features, including manipulation of stored msgs on server HTTP(S): Gmail, Yahoo! Mail, etc. 16

17 POP3 protocol authorization phase v v client commands: user: declare username pass: password server responses +OK -ERR transaction phase, client: v v v v list: list message numbers retr: retrieve message by number dele: delete quit S: +OK POP3 server ready C: user bob S: +OK C: pass hungry C: list S: S: S:. C: retr 1 S: <message 1 contents> S:. C: dele 1 C: retr 2 S: <message 1 contents> S:. C: dele 2 C: quit Self Study S: +OK user successfully logged on S: +OK POP3 server signing off 17

18 POP3 (more) and IMAP Self Study more about POP3 v previous example uses POP3 download and delete mode Bob cannot re-read e- mail if he changes client v POP3 download-andkeep : copies of messages on different clients v POP3 is stateless across sessions IMAP v keeps all messages in one place: at server v allows user to organize messages in folders v keeps user state across sessions: names of folders and mappings between message IDs and folder name 18

19 Quiz: HTTP vs SMTP v Which of the following is not true? A. HTTP is pull-based, SMTP is push-based B. HTTP uses a separate header for each object, SMTP uses a multipart message format C. SMTP uses persistent connections D. HTTP uses client-server communication but SMTP does not 19

20 2. Application Layer: outline 2.1 principles of network applications app architectures app requirements 2.2 Web and HTTP 2.3 FTP 2.4 electronic mail SMTP, POP3, IMAP 2.5 DNS 2.6 P2P applications 2.7 socket programming with UDP and TCP A nice overview 20

21 DNS: domain name system people: many identifiers: TFN, name, passport # Internet hosts, routers: IP address (32 bit) - used for addressing datagrams name, e.g., - used by humans Q: how to map between IP address and name, and vice versa? Domain Name System: v distributed database implemented in hierarchy of many name servers v application-layer protocol: hosts, name servers communicate to resolve names (address/name translation) note: core Internet function, implemented as applicationlayer protocol complexity at network s edge 21

22 DNS: History v Initially all host-address mappings were in a hosts.txt file (in /etc/ hosts): Maintained by the Stanford Research Institute (SRI) Changes were submitted to SRI by New versions of hosts.txt periodically FTP d from SRI An administrator could pick names at their discretion v As the Internet grew this system broke down: SRI couldn t handle the load; names were not unique; hosts had inaccurate copies of hosts.txt v The Domain Name System (DNS) was invented to fix this Jon Postel 22

23 Example use of DNS v v v v You type in the URL window of your web browser Your browser must establish a TCP connection with the CSE web server To do this, your browser needs to know the IP address of the CSE web server How does it obtain the IP address?? The browser passes on the hostname to the client side of the DNS application running on your machine (gethostbyname() function in UNIX) The DNS client sends out a query for mapping the hostname to an IP address into the DNS hierarchy black box over UDP (destination port number: 53) The DNS client receives a reply with the IP address for The browser can now initiate a TCP connection with the HTTP server process located at that IP address 23

24 DNS: services, structure DNS services v hostname to IP address translation v host aliasing canonical, alias names v mail server aliasing v load distribution replicated Web servers: many IP addresses correspond to one name Content Distribution Networks: use IP address of requesting host to find best suitable server Example: closest, leastloaded, etc why not centralize DNS? v single point of failure v traffic volume v distant centralized database v maintenance A: doesn t scale! 24

25 Goals v No naming conflicts (uniqueness) v Scalable many names (secondary) frequent updates v Distributed, autonomous administration Ability to update my own (machines ) names Don t have to track everybody s updates v Highly available v Lookups should be fast 25

26 Key idea: Hierarchy Three intertwined hierarchies Hierarchical namespace As opposed to original flat namespace Hierarchically administered As opposed to centralised (Distributed) hierarchy of servers As opposed to centralised storage 26

27 Hierarchical Namespace root edu com gov mil org net uk fr berkeley eecs sims instr ucla v Top Level Domains are at the top v Domains are sub-trees E.g:.edu, berkeley.edu, eecs.berkeley.edu v Name is leaf-to-root path instr.eecs.berkeley.edu v Depth of tree is arbitrary (limit 128) v Name collisions trivially avoided each domain is responsible 27

28 Hierarchical Administration root edu com gov mil org net uk fr berkeley ucla eecs instr sims A zone corresponds to an administrative authority that is responsible for that portion of the hierarchy E.g., UCB controls names: *.berkeley.edu and *.sims.berkeley.edu v E.g., EECS controls names: *.eecs.berkeley.edu 28

29 Server Hierarchy v Top of hierarchy: Root servers Location hardwired into other servers v Next Level: Top-level domain (TLD) servers.com,.edu, etc. Managed professionally v Bottom Level: Authoritative DNS servers Actually store the name-to-address mapping Maintained by the corresponding administrative authority 29

30 Server Hierarchy v Each server stores a (small!) subset of the total DNS database v An authoritative DNS server stores resource records for all DNS names in the domain that it has authority for v Each server needs to know other servers that are responsible for the other portions of the hierarchy Every server knows the root Root server knows about all top-level domains 30

31 DNS Root v Located in Virginia, USA v How do we make the root scale? Verisign, Dulles, VA 31

32 DNS Root Servers v 13 root servers (labeled A-M; see E NASA Mt View, CA F Internet Software Consortium Palo Alto, CA A Verisign, Dulles, VA C Cogent, Herndon, VA D U Maryland College Park, MD G US DoD Vienna, VA H ARL Aberdeen, MD J Verisign K RIPE London I Autonomica, Stockholm M WIDE Tokyo B USC-ISI Marina del Rey, CA L ICANN Los Angeles, CA 32

33 DNS Root Servers l 13 root servers (labeled A-M; see l Replicated via any-casting E NASA Mt View, CA F Internet Software Consortium, Palo Alto, CA (and 37 other locations) A Verisign, Dulles, VA C Cogent, Herndon, VA (also Los Angeles, NY, Chicago) D U Maryland College Park, MD G US DoD Vienna, VA H ARL Aberdeen, MD J Verisign (21 locations) K RIPE London (plus 16 other locations) I Autonomica, Stockholm (plus 29 other locations) M WIDE Tokyo plus Seoul, Paris, San Francisco B USC-ISI Marina del Rey, CA L ICANN Los Angeles, CA Root Server health: 33

34 Anycast in a nutshell v Routing finds shortest paths to destination v If several locations are given the same address, then the network will deliver the packet to the closest location with that address v This is called anycast Very robust Requires no modification to routing algorithms 34

35 TLD, authoritative servers top-level domain (TLD) servers: responsible for com, org, net, edu, aero, jobs, museums, and all top-level country domains, e.g.: uk, fr, ca, jp Network Solutions maintains servers for.com TLD Educause for.edu TLD authoritative DNS servers: organization s own DNS server(s), providing authoritative hostname to IP mappings for organization s named hosts can be maintained by organization or service provider 35

36 Local DNS name server v does not strictly belong to hierarchy v each ISP (residential ISP, company, university) has one also called default name server v Hosts configured with local DNS server address (e.g., /etc/ resolv.conf) or learn server via a host configuration protocol (e.g., DHCP) v Client application Obtain DNS name (e.g., from URL) Do gethostbyname() to trigger DNS request to its local DNS server v when host makes DNS query, query is sent to its local DNS server has local cache of recent name-to-address translation pairs (but may be out of date!) acts as proxy, forwards query into hierarchy 36

37 DNS name resolution example root DNS server v host at wagner.cse.unsw.edu.au wants IP address for gaia.cs.umass.edu iterated query: v contacted server replies with name of server to contact v I don t know this name, but ask this server local DNS server cse.unsw.edu.au requesting host wagner.cse.unsw.edu.au TLD DNS server 6 authoritative DNS server dns.cs.umass.edu gaia.cs.umass.edu 37

38 DNS name resolution example root DNS server recursive query: v puts burden of name resolution on contacted name server local DNS server cse.unsw.edu.au 5 4 TLD DNS server 1 8 requesting host wagner.cse.unsw.edu.au authoritative DNS server dns.cs.umass.edu gaia.cs.umass.edu Applet: 38

39 Quiz: Which one would you use? Why? root DNS server Recursive queries Iterated queries TLD DNS server 4 root DNS server local DNS server cse.unsw.edu.au local DNS server cse.unsw.edu.au requesting host wagner.cse.unsw.edu.au authoritative DNS server dns.cs.umass.edu requesting host wagner.cse.unsw.edu.au gaia.cs.umass.edu gaia.cs.umass.edu 39

40 DNS: caching, updating records v once (any) name server learns mapping, it caches mapping cache entries timeout (disappear) after some time (TTL) TLD servers typically cached in local name servers thus root name servers not often visited v Subsequent requests need not burden DNS v cached entries may be out-of-date (best effort name-to-address translation!) if name host changes IP address, may not be known Internet-wide until all TTLs expire 40

41 Quiz: DNS Record TTL v The TTL value should be: A. Short, to make sure that changes are accurately reflected B. Long to avoid re-queries of higher level DNS servers C. Something else 41

42 DNS records DNS: distributed db storing resource records (RR) RR format: (name, value, type, ttl) type=a name is hostname value is IP address type=ns name is domain (e.g., foo.com) value is hostname of authoritative name server for this domain type=cname name is alias name for some canonical (the real) name is really servereast.backup2.ibm.com value is canonical name type=mx value is name of mailserver associated with name 42

43 DNS protocol, messages v query and reply messages, both with same message format 2 bytes 2 bytes msg header v v identification: 16 bit # for query, reply to query uses same # flags: query or reply recursion desired recursion available reply is authoritative identification flags # questions # answer RRs # authority RRs # additional RRs questions (variable # of questions) answers (variable # of RRs) authority (variable # of RRs) additional info (variable # of RRs) 43

44 DNS protocol, messages 2 bytes 2 bytes identification # questions flags # answer RRs name, type fields for a query RRs in response to query records for authoritative servers additional helpful info that may be used # authority RRs # additional RRs questions (variable # of questions) answers (variable # of RRs) authority (variable # of RRs) additional info (variable # of RRs) 44

45 bash-3.2$ dig ; <<>> DiG 9.6-ESV-R4 <<>> ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 2, ADDITIONAL: 4 Your 2nd lab ;; QUESTION SECTION: ; IN A ;; ANSWER SECTION: IN CNAME albeniz.orchestra.cse.unsw.edu.au. albeniz.orchestra.cse.unsw.edu.au IN A ;; AUTHORITY SECTION: orchestra.cse.unsw.edu.au IN NS beethoven.orchestra.cse.unsw.edu.au. orchestra.cse.unsw.edu.au IN NS maestro.orchestra.cse.unsw.edu.au. ;; ADDITIONAL SECTION: maestro.orchestra.cse.unsw.edu.au IN A beethoven.orchestra.cse.unsw.edu.au IN A beethoven.orchestra.cse.unsw.edu.au IN A beethoven.orchestra.cse.unsw.edu.au IN A ;; Query time: 1 msec ;; SERVER: #53( ) ;; WHEN: Mon Mar 18 12:36: ;; MSG SIZE rcvd:

46 Inserting records into DNS v example: new startup Network Utopia v register name networkuptopia.com at DNS registrar (e.g., Network Solutions) provide names, IP addresses of authoritative name server (primary and secondary) registrar inserts two RRs into.com TLD server: (networkutopia.com, dns1.networkutopia.com, NS) (dns1.networkutopia.com, , A) v create authoritative server type A record for type MX record for networkutopia.com v Q: Where do you insert these type A and type MX records? Application Layer 46

47 Reliability v DNS servers are replicated (primary/secondary) Name service available if at least one replica is up Queries can be load-balanced between replicas v Usually, UDP used for queries Need reliability: must implement this on top of UDP Spec supports TCP too, but not always implemented v Try alternate servers on timeout Exponential backoff when retrying same server v Same identifier for all queries Don t care which server responds 47

48 DNS provides Indirection v Addresses can change underneath Move to Humans/Apps should be unaffected v Name could map to multiple IP addresses Enables Load-balancing Reducing latency by picking nearby servers v Multiple names for the same address E.g., many services (mail, www, ftp) on same machine E.g., aliases like and cnn.com v But, this flexibility applies only within domain! 48

49 DNS Load Balancing (Authoritative) DNS server monitors the load of the multiple web servers Replies back with the IP address for the least loaded server Can also consider the location of the client (e.g., a user from Sydney directed to Australian server) 49

50 DNS for Content Distribution 0: cnn.com distributes images to Akamai server 1: An end user types in in their browser 2: index page contains link to the a73.g.akamai.net 3: local DNS server asks root DNS server to resolve a73.g.akamai.net 4: root DNS replies with IP address of high-level Akamai DNS server, akamai.net 5: local DNS server queries this nameserver to resolve a73.g.akamai.net 6: high-level DNS server replies with IP address of the DNS server for g.akamai.net 7: local DNS queries this nameserver to resolve a73.g.akamai.net 8: low level DNS server takes into consideration the client s location and replies with the IP address of the closest Akamai web server that hosts the image 9: end user s machine sends a GET request for the image to this closest Akamai web server 10: Akamai webserver responds with the image 50

51 Reverse DNS v IP address -> domain name v Special PTR record type to store reverse DNS entries v Where is reverse DNS used? Troubleshooting tools such as traceroute and ping Received trace header field in SMTP SMTP servers for validating IP addresses of originating servers Internet forums tracking users System logging or monitoring tools Used in load balancing servers/content distribution to determine location of requester 51

52 Do you trust your DNS server? v Censorship v Logging IP address, websites visited, geolocation data and more E.g., Google DNS: 52

53 Attacking DNS DDoS attacks v Bombard root servers with traffic Not successful to date Traffic Filtering Local DNS servers cache IPs of TLD servers, allowing root server to be bypassed v Bombard TLD servers Potentially more dangerous Redirect attacks v Man-in-middle Intercept queries v DNS poisoning Send bogus replies to DNS server, which caches Exploit DNS for DDoS v Send queries with spoofed source address: target IP v Requires amplification Want to dig deeper? 53

54 DNS Cache Poisoning v Suppose you are a bad guy and you control the name server for drevil.com. Your name server receives a request to resolve and you respond as follows: ;; QUESTION SECTION: ; IN A ;; ANSWER SECTION: IN CNAME ;; AUTHORITY SECTION: drevil.com IN NS dns1.drevil.com. drevil.com IN NS google.com ;; ADDITIONAL SECTION: google.com 600 IN A A drevil.com machine, not google.com v Solution: Do not allow DNS servers to cache IP address mappings unless they are from authoritative name servers 54

55 Dig deeper? DNS Cache Poisoning Test DNSSEC: DNS Security Extensions, 55

56 2. Application Layer: outline 2.1 principles of network applications app architectures app requirements 2.2 Web and HTTP 2.3 FTP 2.4 electronic mail SMTP, POP3, IMAP 2.5 DNS 2.6 P2P applications 2.7 socket programming with UDP and TCP 56

57 Pure P2P architecture v no always-on server v arbitrary end systems directly communicate v peers are intermittently connected and change IP addresses examples: file distribution (BitTorrent) Streaming (KanKan) VoIP (Skype) 57

58 File distribution: client-server vs P2P Question: how much time to distribute file (size F) from one server to N peers? peer upload/download capacity is limited resource u s : server upload capacity file, size F server u s u 1 d 1 u 2 d 2 d i : peer i download capacity u N d N network (with abundant bandwidth) d i u i u i : peer i upload capacity 58

59 File distribution time: client-server v server transmission: must sequentially send (upload) N file copies: time to send one copy: F/u s time to send N copies: NF/u s v client: each client must download file copy d min = min client download rate min client download time: F/d min F u s network d i u i time to distribute F to N clients using client-server approach D c-s > max{nf/u s,,f/d min } increases linearly in N 59

60 File distribution time: P2P v server transmission: must upload at least one copy time to send one copy: F/u s v client: each client must download file copy min client download time: F/d min v clients: as aggregate must download NF bits F u s network d i u i max upload rate (limiting max download rate) is u s + Σu i time to distribute F to N clients using P2P approach D P2P > max{f/u s,,f/d min,,nf/(u s + Σu i )} increases linearly in N but so does this, as each peer brings service capacity 60

61 Client-server vs. P2P: example client upload rate = u, F/u = 1 hour, u s = 10u, d min u s Minimum Distribution Time P2P Client-Server N 61

62 P2P file distribution: BitTorrent v file divided into 256Kb chunks v peers in torrent send/receive file chunks tracker: tracks peers participating in torrent torrent: group of peers exchanging chunks of a file Alice arrives obtains list of peers from tracker and begins exchanging file chunks with peers in torrent 62

63 .torrent files v Contains address of trackers for the file Where can I find other peers? v Contain a list of file chunks and their cryptographic hashes This ensures that chunks are not modified 63

64 P2P file distribution: BitTorrent v peer joining torrent: has no chunks, but will accumulate them over time from other peers registers with tracker to get list of peers, connects to subset of peers ( neighbours ) v while downloading, peer uploads chunks to other peers v peer may change peers with whom it exchanges chunks v churn: peers may come and go v once peer has entire file, it may (selfishly) leave or (altruistically) remain in torrent 64

65 BitTorrent: requesting, sending file chunks requesting chunks: v at any given time, different peers have different subsets of file chunks v periodically, Alice asks each peer for list of chunks that they have v Alice requests missing chunks from peers, rarest first sending chunks: tit-for-tat v Alice sends chunks to those four peers currently sending her chunks at highest rate other peers are choked by Alice (do not receive chunks from her) re-evaluate top 4 every10 secs v every 30 secs: randomly select another peer, starts sending chunks optimistically unchoke this peer newly chosen peer may join top 4 65

66 BitTorrent: tit-for-tat (1) Alice optimistically unchokes Bob (2) Alice becomes one of Bob s top-four providers; Bob reciprocates (3) Bob becomes one of Alice s top-four providers higher upload rate: find better trading partners, get file faster! Original Research Paper on BitTorrent linked to lecture notes page 66

67 Getting rid of the server v Distribute the tracker information using a Distributed Hash Table (DHT) v A DHT is a lookup structure Maps keys to an arbitrary value Works a lot like, well. hash table Title House of Cards Season 4 Walking Dead Season 6 Game of Thrones Season Better Call Saul Season 2 End-systems that have it Michonne, Rick, Chuck, Sansa Frank, Mike, Tyrion, Gregor Saul, Mike, Glenn Khal, Jon, Daryl 67

68 Getting rid of the server v Distribute the tracker information using a Distributed Hash Table (DHT) v A DHT is a lookup structure Maps keys to an arbitrary value Works a lot like, well. hash table Title End-systems that have it House of Cards Season , Walking Dead Season , Game of Thrones Season Better Call Saul Season

69 Recall: Hash Function v Mapping of any data to an integer E.g., md5sum, sha1, etc. md5: 04c3416cadd85971a129dd1de86cee49 v With a good (cryptographic) hash Hash values likely to be unique, although duplicates are possible Very difficult to find collisions (hashes spread out) 69

70 Recall: Hash Table v N buckets v Key-value pair is assigned bucket i i = HASH(key)%N v Easy to look up value based on key v Multiple key-value pairs assigned to each bucket 70

71 Distributed Hash Table (DHT) v DHT: a distributed P2P database v database has (key, value) pairs; examples: key: TFN number; value: human name key: file name; value: BT tracker peer(s) v Distribute the (key, value) pairs over the (millions of peers) v a peer queries DHT with key DHT returns values that match the key v peers can also insert (key, value) pairs 71

72 Overlay network v A network made up of virtual or logical links v Virtual links map to one or more physical links (i.e. paths) 72

73 Challenges v How do we assign (key, value) pairs to nodes? v How do we find them again quickly? v What happens if nodes join/leave? 73

74 Q: how to assign keys to peers? v basic idea: convert each key to an integer Assign integer to each peer put (key,value) pair in the peer that is closest to the key 74

75 DHT identifiers: Consistent Hashing v assign integer identifier to each peer in range [0,2 n -1] for some n-bit hash function E.g., node ID is hash of its IP address v require each key to be an integer in same range v to get integer key, hash original key e.g., key = hash( House of Cards Season 4 ) this is why its is referred to as a distributed hash table 75

76 Assign keys to peers v rule: assign key to the peer that has the closest ID. v common convention: closest is the immediate successor of the key. v e.g., n=4; peers: 1,3,4,5,8,10,12,14; key = 13, then successor peer = 14 key = 15, then successor peer = 1 76

77 Circular DHT (1) v each peer only aware of immediate successor and predecessor. v overlay network 77

78 Circular DHT (2) 1111 I am Who s responsible for key 1110? Define closest as closest successor

79 Quiz: DHT Complexity v Given N nodes, what is the complexity (number of messages) of finding a value when each peer knows its successor? A. O (log n) B. O (n) C. O (n 2 ) D. O (2 n ) 79

80 Circular DHT with shortcuts Who s responsible for key 1110? v each peer keeps track of IP addresses of predecessor, successor, short cuts v reduced from 6 to 2 messages. v possible to design shortcuts so O(log N) neighbours, O(log N) messages in query 5 80

81 Peer churn example: peer 5 abruptly leaves handling peer churn: v peers may come and go (churn) v each peer knows address of its two successors v each peer periodically pings its two successors to check aliveness v if immediate successor leaves, choose next successor as new immediate successor v peer 4 detects peer 5 departure; makes 8 its immediate successor; asks 8 who its immediate successor is; makes 8 s immediate successor its second successor. 81

82 1 Quiz: Peer Churn v What happens to 5 s value? A. Lost forever B. Lost until 5 comes back Still recoverable (how?) 82

83 Quiz: DHT v Which of the following is not true? A. DHT distributes portions of a hash table across peers. B. The key corresponding to an object (e.g., movie) depends on the current number of peers. C. Which peer is responsible for an object depends on the current number of peers. 83

84 More DHT info v How do nodes join? v How does cryptographic hashing work? v How much state does each node store? Research Papers (on the lectures page): Chord: A Scalable Peer-to-Peer Lookup Service for Internet Applications Dynamo: Amazon s Highly Available Key-value Store 84

85 DHT: Applications v File sharing and backup [CFS, Ivy, OceanStore, PAST, Pastiche ] v Web cache and replica [Squirrel, Croquet Media Player] v Censor-resistant stores [Eternity] v DB query and indexing [PIER, Place Lab, VPN Index] v Event notification [Scribe] v Naming systems [ChordDNS, Twine, INS, HIP] v Distributed BitTorrent tracker [Kademlia, Vuze] v Communication primitives [I3, ] v Host mobility [DTN Tetherless Architecture] 85

86 2. Application Layer: outline 2.1 principles of network applications app architectures app requirements 2.2 Web and HTTP 2.3 FTP 2.4 electronic mail SMTP, POP3, IMAP 2.5 DNS 2.6 P2P applications 2.7 socket programming with UDP and TCP Note: 6 th Edition uses Python. Lecture notes show examples of both Java and Python 86

87 Socket programming Self Study goal: learn how to build client/server applications that communicate using sockets socket: door between application process and endend-transport protocol application process socket application process controlled by app developer transport network link physical Internet transport network link physical controlled by OS 87

88 Socket programming Self Study Two socket types for two transport services: UDP: unreliable datagram TCP: reliable, byte stream-oriented Application Example: 1. Client reads a line of characters (data) from its keyboard and sends the data to the server. 2. The server receives the data and converts characters to uppercase. 3. The server sends the modified data to the client. 4. The client receives the modified data and displays the line on its screen. 88

89 Socket programming with UDP Self Study UDP: no connection between client & server v no handshaking before sending data v sender explicitly attaches IP destination address and port # to each packet v rcvr extracts sender IP address and port# from received packet UDP: transmitted data may be lost or received out-of-order Application viewpoint: v UDP provides unreliable transfer of groups of bytes ( datagrams ) between client and server 89

90 Self Study Client/server socket interaction: UDP (Java) Server (running on hostid) Client create socket, port=x, for incoming request: serversocket = DatagramSocket() read request from serversocket write reply to serversocket specifying client host address, port number create socket, clientsocket = DatagramSocket() Create, address (hostid, port=x, send datagram request using clientsocket read reply from clientsocket close clientsocket 90

91 Example: Java client (UDP) Self Study import java.io.*; import java.net.*; Create input stream Create client socket Translate hostname to IP address using DNS class UDPClient { public static void main(string args[]) throws Exception { BufferedReader infromuser = new BufferedReader(new InputStreamReader(System.in)); DatagramSocket clientsocket = new DatagramSocket(); InetAddress IPAddress = InetAddress.getByName("hostname"); byte[] senddata = new byte[1024]; byte[] receivedata = new byte[1024]; String sentence = infromuser.readline(); senddata = sentence.getbytes(); 91

92 Example: Java client (UDP), cont. Self Study Create datagram with data-to-send, length, IP addr, port Send datagram to server Read datagram from server DatagramPacket sendpacket = new DatagramPacket(sendData, senddata.length, IPAddress, 9876); clientsocket.send(sendpacket); DatagramPacket receivepacket = new DatagramPacket(receiveData, receivedata.length); clientsocket.receive(receivepacket); String modifiedsentence = new String(receivePacket.getData()); System.out.println("FROM SERVER:" + modifiedsentence); clientsocket.close(); } } 92

93 Example: Java server (UDP) Self Study import java.io.*; import java.net.*; Create datagram socket at port 9876 class UDPServer { public static void main(string args[]) throws Exception { DatagramSocket serversocket = new DatagramSocket(9876); byte[] receivedata = new byte[1024]; byte[] senddata = new byte[1024]; Create space for received datagram Receive datagram while(true) { DatagramPacket receivepacket = new DatagramPacket(receiveData, receivedata.length); serversocket.receive(receivepacket); 93

94 Example: Java server (UDP), cont Self Study String sentence = new String(receivePacket.getData()); Get IP addr port #, of sender InetAddress IPAddress = receivepacket.getaddress(); int port = receivepacket.getport(); String capitalizedsentence = sentence.touppercase(); senddata = capitalizedsentence.getbytes(); Create datagram to send to client Write out datagram DatagramPacket sendpacket = new DatagramPacket(sendData, senddata.length, IPAddress, port); serversocket.send(sendpacket); to socket } } } End of while loop, loop back and wait for another datagram 94

95 Client/server socket interaction: UDP (Python) Self Study server (running on serverip) create socket, port= x: serversocket = socket(af_inet,sock_dgram) read datagram from serversocket write reply to serversocket specifying client address, port number client create socket: clientsocket = socket(af_inet,sock_dgram) Create datagram with server IP and port=x; send datagram via clientsocket read datagram from clientsocket close clientsocket 95

96 Example app: UDP client Self Study include Python s socket library create UDP socket for server get user keyboard input Attach server name, port to message; send into socket read reply characters from socket into string print out received string and close socket Python UDPClient from socket import * servername = hostname serverport = clientsocket = socket(socket.af_inet, socket.sock_dgram) message = raw_input( Input lowercase sentence: ) clientsocket.sendto(message,(servername, serverport)) modifiedmessage, serveraddress = clientsocket.recvfrom(2048) print modifiedmessage clientsocket.close() 96

97 Example app: UDP server Self Study Python UDPServer create UDP socket bind socket to local port number loop forever Read from UDP socket into message, getting client s address (client IP and port) send upper case string back to this client from socket import * serverport = serversocket = socket(af_inet, SOCK_DGRAM) serversocket.bind(('', serverport)) print The server is ready to receive while 1: message, clientaddress = serversocket.recvfrom(2048) modifiedmessage = message.upper() serversocket.sendto(modifiedmessage, clientaddress) 97

98 Socket programming with TCP Self Study client must contact server v server process must first be running v server must have created socket (door) that welcomes client s contact client contacts server by: v Creating TCP socket, specifying IP address, port number of server process v when client creates socket: client TCP establishes connection to server TCP v when contacted by client, server TCP creates new socket for server process to communicate with that particular client allows server to talk with multiple clients source port numbers used to distinguish clients (more in Chap 3) application viewpoint: TCP provides reliable, in-order byte-stream transfer ( pipe ) between client and server 98

99 TCP Sockets Self Study 99

100 Client/server socket interaction: TCP (in Java) Self Study Server (running on hostid) create socket, port=x, for incoming request: welcomesocket = ServerSocket() wait for incoming connection request connectionsocket = welcomesocket.accept() read request from connectionsocket write reply to connectionsocket close connectionsocket TCP connection setup Client create socket, connect to hostid, port=x clientsocket = Socket() send request using clientsocket read reply from clientsocket close clientsocket 100

101 Example: Java client (TCP) Self Study import java.io.*; import java.net.*; class TCPClient { public static void main(string argv[]) throws Exception { String sentence; String modifiedsentence; Create input stream Create client socket, connect to server Create output stream attached to socket BufferedReader infromuser = new BufferedReader(new InputStreamReader(System.in)); Socket clientsocket = new Socket("hostname", 6789); DataOutputStream outtoserver = new DataOutputStream(clientSocket.getOutputStream()); 101

102 Self Study Example: Java client (TCP), cont. Create input stream attached to socket BufferedReader infromserver = new BufferedReader(new InputStreamReader(clientSocket.getInputStream())); sentence = infromuser.readline(); Send line to server Read line from server outtoserver.writebytes(sentence + '\n'); modifiedsentence = infromserver.readline(); System.out.println("FROM SERVER: " + modifiedsentence); } } clientsocket.close(); 102

103 Example: Java server (TCP) import java.io.*; Self Study import java.net.*; class TCPServer { Create welcoming socket at port 6789 Wait, on welcoming socket for contact by client Create input stream, attached to socket public static void main(string argv[]) throws Exception { String clientsentence; String capitalizedsentence; ServerSocket welcomesocket = new ServerSocket(6789); while(true) { Socket connectionsocket = welcomesocket.accept(); BufferedReader infromclient = new BufferedReader(new InputStreamReader(connectionSocket.getInputStream())); 103

104 Self Study Example: Java server (TCP), cont Create output stream, attached to socket Read in line from socket DataOutputStream outtoclient = new DataOutputStream(connectionSocket.getOutputStream()); clientsentence = infromclient.readline(); Write out line to socket } } } capitalizedsentence = clientsentence.touppercase() + '\n'; outtoclient.writebytes(capitalizedsentence); End of while loop, loop back and wait for another client connection 104

105 Example app: TCP client Self Study Python TCPClient create TCP socket for server, remote port No need to attach server name, port from socket import * servername = servername serverport = clientsocket = socket(af_inet, SOCK_STREAM) clientsocket.connect((servername,serverport)) sentence = raw_input( Input lowercase sentence: ) clientsocket.send(sentence) modifiedsentence = clientsocket.recv(1024) print From Server:, modifiedsentence clientsocket.close() 105

106 Example app: TCP server Self Study Python TCPServer create TCP welcoming socket server begins listening for incoming TCP requests loop forever server waits on accept() for incoming requests, new socket created on return from socket import * serverport = serversocket = socket(af_inet,sock_stream) serversocket.bind((,serverport)) serversocket.listen(1) print The server is ready to receive while 1: connectionsocket, addr = serversocket.accept() read bytes from socket (but not address as in UDP) close connection to this client (but not welcoming socket) sentence = connectionsocket.recv(1024) capitalizedsentence = sentence.upper() connectionsocket.send(capitalizedsentence) connectionsocket.close() 106

107 Application Layer: summary our study of network apps now complete! v application architectures client-server P2P v application service requirements: reliability, bandwidth, delay v Internet transport service model connection-oriented, reliable: TCP unreliable, datagrams: UDP v specific protocols: HTTP FTP SMTP, POP, IMAP DNS P2P: BitTorrent, DHT v socket programming: TCP, UDP sockets 107

108 Application Layer: summary most importantly: learned about protocols! v typical request/reply message exchange: client requests info or service server responds with data, status code v message formats: headers: fields giving info about data data: info being communicated important themes: v control vs. data msgs in-band, out-of-band v centralized vs. decentralized v stateless vs. stateful v reliable vs. unreliable msg transfer v complexity at network edge Solve all sample problems + those posted with lectures Reading Exercise for next week: Chapter 3: