Network Programming. Copyright 2017 by Robert M. Dondero, Ph.D. Princeton University

Transcription

1 Network Programming Copyright 2017 by Robert M. Dondero, Ph.D. Princeton University 1

2 Objectives (Ideal) You will learn about: Network basics The Internet Network programming in Java and Python How to compose a client How to compose a (multithreaded) server How to serialize and communicate objects 2

3 Agenda (Ideal) Network basics The Internet Network programming Communicating objects 3

4 Objectives You will learn about: Network programming in Java and Python How to compose a client How to compose a (multithreaded) server How to serialize and communicate objects Network basics (as time allows) The Internet (as time allows) 4

5 Agenda Network programming Communicating objects Network basics (as time allows) The Internet (as time allows) 5

6 Key Concept: Network Address Any computer on the Internet has two addresses: Medium Access Control (MAC) address Used by network adapters, bridges, routers Example: 90:1b:0e:6a:32:26 Internet Protocol (IP) address Used by routers (and people) 32 bit integer, usually expressed in dotted-decimal form Example: Example:

7 Key Concept: Network Address Many computers on the Internet have a third address: Domain name Used by people Domain Name System (DNS) converts to IP address Example: courselab01.cs.princeton.edu Example: localhost 7

8 Key Concept: Port Port A software abstraction 16-bit integer Identifies unique process on specified host Client process on host X communicates with a server process on host Y via ports Server uses a known port on server Client uses an ephemeral port 8

9 Key Concept: Socket Socket IP address + port Dominant abstraction in network programming First developed by Unix/C community Beautiful abstraction Adopted by Java, Python, many others 9

10 Aside: Sockets and Unix File descriptor table File table File table is a surrogate for: the terminal (/dev/tty) a file Process Process can use same functions to read from or write to terminal or file Process can use Unix system-level functions (read, write) Process can wrap FILE objects around file tables, and use standard I/O functions (scanf, printf, ) 10

11 Aside: Sockets and Unix File descriptor table File table File table is a surrogate for: the terminal (/dev/tty) a file a pipe Process Process can use same functions to read from or write to terminal or file or another process on same computer We ll study pipes later 11

12 Aside: Sockets and Unix File descriptor table Process File table File table is a surrogate for: the terminal (/dev/tty) a file a pipe a socket Process can use same functions to read from or write to terminal or file or another process on same computer or another process on different computer We ll study sockets now 12

13 Client and Server Sockets courselab01 Client Process listen Known Port courselab02 Server Process Server Socket Server process creates ServerSocket on host courselab02.cs.princeton.edu port ServerSocket listens for connection requests 13

14 Client and Server Sockets courselab01 Client Process Socket Ephemeral Port connection request Known Port courselab02 Server Process Server Socket Client process creates Socket on host courselab01.cs.princeton.edu (port 50830) Client process requests connection to server process at host courselab02.cs.princeton.edu port

15 Client and Server Sockets courselab01 Client Process Socket Ephemeral Port accept Known Port courselab02 Server Process Server Socket Socket Server process accepts request ServerSocket creates Socket on host courselab02.cs.princeton.edu port ServerSocket informs client process of Socket 15

16 Client and Server Sockets courselab01 Client Process Socket Ephemeral Port read/write Known Port courselab02 Server Process Server Socket Socket Client process reads from or writes to socket on host courselab02.cs.princeton.edu port Server processes reads from or writes to socket on host courselab01.cs.princeton.edu at port

17 Network Pgmming in COS 333 The ideal: Teammate1 Local Computer Client Teammate2 Local Computer Client socket socket CourseLab Firewall courselab01/02 Server (Teammate1) Problem: firewall 17

18 Network Pgmming in COS 333 Option 1: Run on CourseLab CourseLab Firewall courselab02 Client (Teammate1) socket courselab01 Server (Teammate1) Client (Teammate2) socket Important: Use ports in range

19 Network Pgmming in COS 333 Option 2: Run on local computers Teammate2 Local Computer Client socket Teammate1 Local Computer Server Client socket Teammate1 issues ifconfig (Mac/Linux) or ipconfig (MS Windows) command to determine IP addr of server host 19

20 Aside: Telnet Telnet program Provides access to a command-line interface on a remote host Similar to ssh But without data encryption Often useful as a client when testing server programs 20

21 DayTime Example Client: Connects to server Server: Accepts connection Writes current day and time to client Disconnects Client: Reads current day and time from server Writes to stdout 21

22 DayTime in Java See DayTimeClient.java and DayTimeServer.java Try: telnet time-a.nist.gov 13 java DayTimeClient time-a.nist.gov 13 Try: java DayTimeServer someport telnet localhost someport Try: java DayTimeServer someport java DayTimeClient localhost someport 22

23 DayTime in Python See daytimeclient.py and daytimeserver.py AF_INET => IP connection SOCK_STREAM => TCP connection [SOCK_DGRAM => UDP (datagram) connection] 23

24 DayTime in Python See daytimeclient.py and daytimeserver.py (cont.) Try: telnet time-a.nist.gov 13 python daytimeclient.py time-a.nist.gov 13 Try: python daytimeserver.py someport telnet localhost someport Try: python daytimeserver.py someport python daytimeclient.py localhost someport 24

25 Echo Example Client: Connects to server, reads line from stdin, writes line to server Server: Reads line from client, writes uppercase line to client Client: Reads echoed line from server, writes echoed line to stdout, repeats until EOF of stdin Note: Sockets are bi-directional 25

26 Echo in Java See EchoServer.java and EchoClient.java Try: java EchoServer someport telnet localhost someport Try: java EchoServer someport java EchoClient localhost someport 26

27 Echo in Java Problem: EchoServer.java can handle only 1 client at a time Try: java EchoServer someport telnet localhost someport telnet localhost someport Try: java EchoServer someport java EchoClient localhost someport java EchoClient localhost someport 27

28 Echo in Java Solution: See EchoServerMult.java Multithreaded server Described more thoroughly later in course Try: java EchoServerMult someport telnet localhost someport telnet localhost someport Try: java EchoServerMult someport java EchoClient localhost someport java EchoClient localhost someport 28

29 Echo in Python See echoserver.py and echoclient.py Try: python echoserver.py someport telnet localhost someport Try: python echoserver.py someport python echoclient.py localhost someport 29

30 Echo in Python Problem: echoserver.py can handle only 1 client at a time Try: python echoserver.py someport telnet localhost someport telnet localhost someport Try: python echoserver.py someport python echoclient.py localhost someport python echoclient.py localhost someport 30

31 Echo in Python Solution: See echoservermult.py Multithreaded server Described more thoroughly later in course Try: python echoservermult.py someport telnet localhost someport telnet localhost someport Try: python echoservermult.py someport python echoclient.py localhost someport python echoclient.py localhost someport 31

32 Echo Conclusion Note: EchoServer.java works with: Telnet EchoClient.java echoclient.py echoserver.py works with: Telnet echoclient.py EchoClient.java 32

33 Agenda Network programming Communicating objects Network basics (as time allows) The Internet (as time allows) 33

34 Courses Example Server: Writes Course objects (referencing Book objects) to client Client: Reads Course objects (referencing Book objects) from server Writes them to stdout 34

35 Courses in Java See Book.java See Course.java For now, ignore Serializable 35

36 Courses in Java See CourseServer1.java Convert int course count to a human-readable byte sequence; write to client Convert Course object to a human-readable byte sequence; write to client; repeat course count times See CourseClient1.java Read a human-readable byte sequence from server; convert to int course count Read a human-readable byte sequence from server; convert to Course object; repeat course count times 36

37 Courses in Java Fine But is there an easier way? 37

38 Serializable Interface Serializable Tagging interface; no methods ObjectOutputStream serializes object Converts primitive datum or Serializable object to byte stream ObjectInputStream deserializes object Converts byte stream to primitive datum or Serializable object Data of type int and type double, and objects of class String are Serializable 38

39 Courses in Java See CourseServer2.java Write int course count to client Write Course object s String objects & double to client, repeat course count times See CourseClient2.java Read int course count from server Read String objects & double from server; create Course object; repeat course count times 39

40 Courses in Java But wait! Book objects and Course objects are objects... Can define the Book and Course classes to implement Serializable 40

41 Courses in Java See CourseServer3.java Write int course count to client Write Course object to client; repeat course count times See CourseClient3.java Read int course count from server Read Course object from server; repeat course count times 41

42 Courses in Java But wait! ArrayList objects also are Serializable... 42

43 Courses in Java See CourseServer4.java Write an ArrayList object to server! See CourseClient4.java Read an ArrayList object from server! No need to write/read course count! 43

44 Courses in Python See book.py See course.py 44

45 Courses in Python See courseserver1.py Convert int course count to a human-readable byte sequence; write to client Convert Course object to a human-readable byte sequence; write to client; repeat course count times See courseclient1.py Read a human-readable byte sequence from server; convert to int course count Read a human-readable byte sequence from server; convert to Course object; repeat course count times 45

46 Courses in Python Fine But is there an easier way? 46

47 The Pickle Module pickle.dump(obj, flo) Serializes obj to byte stream Writes byte stream to flo obj = pickle.load(flo) Reads byte stream from flo Deserializes byte stream to form obj 47

48 Courses in Python See courseserver2.py Write int course count to client Write Course object s str objects & float to client, repeat course count times See courseclient2.py Read int course count from server Read str objects & float from server; create Course object; repeat course count times 48

49 Courses in Python But wait! Book objects and Course objects are objects and so can be pickled... 49

50 Courses in Python See courseserver3.py Write int course count to client Write Course object to client; repeat course count times See courseclient3.py Read int course count from server Read Course object from server; repeat course count times 50

51 Courses in Python But wait! list objects are objects, and so can be pickled 51

52 Courses in Python See courseserver4.py Write a list object to client! See courseclient4.py Read a list object from server! No need to write/read course count! 52

53 Courses Conclusion Note: CourseServer1.java works with: CourseClient1.java courseclient1.py telnet! courseserver1.py works with: courseclient1.py CourseClient1.java telnet! 53

54 Courses Conclusion Note: CourseServer2.java works only with CourseClient2.java courseserver2.py works only with courseclient2.py Same for 3 and 4 Object serialization/pickling is languagespecific 54

55 Comm of Shared Objects Problem: What if multiple objects are composed of the same nested objects... 55

56 Comm of Shared Objects Using the Courses example... In server: Course0 Course1 Course2 Book0 Book1 56

57 Comm of Shared Objects In client (bad): Course0 Course1 Course2 Book0 Book1 Book1 No!!! In client (good): Course0 Course1 Course2 Book0 Book1 57

58 Shared Objects in Java Solution: Handled automatically! ObjectOutputStream maps each unique object address to a serial num Sees same object address again => writes only serial num ObjectInputStream maps serial numbers to unique objects Sees same serial num again => creates ref to existing object 58

59 Shared Objects in Java See CourseServer5.java See CourseClient5.java Change of Book price in client confirms that shared object was written/read properly COS 217 COS 226 COS 333 King Sedgewick Kernighan 59

60 Shared Objects in Python Solution: Handled automatically! pickle.dump() maps each unique object address to a serial num Sees same object address again => writes only serial num pickle.load() maps serial numbers to unique objects Sees same serial num again => creates ref to existing object 60

61 Shared Objects in Python See courseserver5.py See courseclient5.py Change of Book price in client confirms that shared object was written/read properly COS 217 COS 226 COS 333 King Sedgewick Kernighan 61

62 Agenda Network programming Communicating objects Network basics (as time allows) The Internet (as time allows) 62

63 Network Basics: Goals Networking Huge topic Can only scratch the surface Goal Convey workable mental model From programmer s point of view Approach Bottom-up... 63

64 Hosts and Network Adapters Host Computer Network Adapter Network Network adapter: HW that connects host computer to network Each has unique "medium access control (MAC) address" MAC address: xx:xx:xx:yy:yy:yy, where each x and y is a hexadecimal digit xx:xx:xx is unique to vendor yy:yy:yy is unique to device 64

65 Hosts and Network Adapters Try on CourseLab: ifconfig Note ether 65

66 Hubs and Segments Host1 Adapter header payload Host2 Adapter header payload Network Segment frame Hub frame Frame = header + payload Header identifies source and destination addresses Payload contains user data Hub (alias repeater) = HW (no SW) that transmits frames among hosts Receives frame from adapter; doesn t examine header; copies to all others Every adapter sees frame; only destination adapter reads it Network Segment = hosts + hub Scope: one room; one floor of a building 66

67 Bridges and LANs Host1 Host2 Hub1 Bridge1 Host3 Hub3 Bridge2 Hub4 LAN Host4 Host5 Host6 Host7 Bridge (alias switch) = HW+SW that connects hosts, hubs, other bridges Does examine headers Analyzes message sources; learns where hosts are host1 host2: hub1 bridge1 (discard) host1 host7: hub1 bridge1 bridge2 hub4 Local Area Network (LAN) = hosts + hubs + bridges Scope: one building; one campus 67

68 Routers and WANs LAN1 LAN2 Router WAN Router (~alias gateway) = HW that connects multiple (incompatible) LANs Wide Area Network (WAN) = LANs + routers Scope: planet Earth! And beyond! Each host has both a LAN (MAC) address and a WAN address LAN address: not related to network topology WAN address: related to network topology 68

69 WAN Data Transfer Host1 Protocol SW on Host1 payload When Host1 on LAN1 sends a payload to Host2 on LAN2... WAN packet LAN1 Adapter on Host1 LAN1 header (MAC addr of Router) WAN header (WAN addr of Host2) LAN1 frame payload Hub Bridge 69

70 WAN Data Transfer WAN packet LAN1 header (MAC addr of Router) WAN header (WAN addr of Host2) payload Router LAN1 Adapter LAN1 frame Protocol SW on Router WAN packet WAN header (WAN addr of Host2) payload 70

71 WAN Data Transfer WAN packet WAN header (WAN addr of Host2) payload Router LAN2 Adapter Protocol SW on Router WAN packet LAN2 header (MAC addr of Host2) WAN header (WAN addr of Host2) payload LAN2 frame 71

72 WAN Data Transfer Bridge Hub LAN2 Adapter on Host2 Protocol SW on Host2 Host2 LAN2 header (MAC addr of Host2) payload WAN header (WAN addr of Host2) LAN2 frame WAN packet payload 72

73 Protocol Stacks Generic Packet Structure Level 0 header Level 1 header Level 2 header... payload Level 2 packet Level 1 packet Level 0 packet Protocols can be (and typically are) layered/stacked And so packets can be (and typically are) layered/stacked 73

74 Agenda Network programming Communicating objects Network basics (as time allows) The Internet (as time allows) 74

75 The Internet The Internet A WAN that uses a particular protocol stack 75

76 The Internet Protocol Stack Application Layer HTTP, HTTPS, SMTP, POP3, IMAP, FTP, SFTP telnet, SSH, daytime, echo, DNS, Transport Layer TCP UDP Internet Layer IP Typical Internet Packet Structure LAN header IP header TCP header App header... payload 76

77 The Internet Layer Application Layer HTTP, HTTPS, SMTP, POP3, IMAP, FTP, SFTP telnet, SSH, daytime, echo, DNS, Transport Layer Internet Layer TCP UDP IP Dominant protocol: IP (Internet Protocol) 77

78 IP Characteristics IP characteristics Connectionless No persistent connection Packetized Sender splits long message into packets Receiver re-assembles the packets Unreliable Packets can be lost (errors, congestion) Receiver not notified of lost packets 78

79 IP Headers IP Header Source WAN address Destination WAN address 79

80 IP Addresses IP addresses WAN address = IP address Internal form: 32 bits Human-readable form: dotted decimal xxx.xxx.xxx.xxx 80

81 Some IP Addresses IP Address Computer Princeton courselab cluster computers Princeton CS Dept web server host computer Princeton web server host computer A US gov computer The local host 81

82 Exploring IP Try on CourseLab: ifconfig Note inet Note: is IP address of the local host Useful for testing networking apps 82

83 Domain Names Problem: Difficult for humans to remember/use dotted decimal IP addresses Solution: Domain names Domain name: A symbolic name for IP address(es) Example: Example: Example: localhost 83

84 The Domain Name System Problem: How to map domain name to IP address(es)? Solution 1: hosts.txt file at SRI International Downloaded ~weekly Didn t scale Solution 2: Domain Name System 84

85 The Domain Name System The Domain Name System Domain name system (DNS) The phone book of the Internet Maps domain names to their IP addresses Distributed hierarchical database 85

86 The Domain Name System Your Computer What is DNS Resolver What is Root Name Server References Try xxx.xx.xxx.x (Addr of edu name server) 86

87 The Domain Name System Your Computer What is DNS Resolver What is edu Name Server References Try yyy.yy.yyy.y (addr of princeton.edu name server) 87

88 The Domain Name System Your Computer It's DNS Resolver What is princeton.edu Name Server References It's

89 DNS root servers: DNS Root Servers Many hundreds; over 130 physical locations 89

90 DNS Root Servers Q: How can DNS root servers handle the heavy workload? A: Caching at each level of the DNS hierarchy In reality, root servers handle mostly silly requests 90

91 Exploring DNS Try on CourseLab: nslookup courselab.cs.princeton.edu nslookup courselab01.cs.princeton.edu nslookup nslookup nslookup time-a.nist.gov nslookup nonexistingdomainname 91

92 Exploring DNS See IpAddress.java java IpAddress courselab.cs.princeton.edu java IpAddress courselab01.cs.princeton.edu java IpAddress java IpAddress java IpAddress time-a.nist.gov java IpAddress localhost java IpAddress nonexistingdomainname 92

93 Exploring DNS See ipaddress.py python ipaddress.py courselab.cs.princeton.edu python ipaddress.py courselab01.cs.princeton.edu python ipaddress.py python ipaddress.py python ipaddress.py time-a.nist.gov python ipaddress.py localhost python ipaddress.py nonexistingdomainname 93

94 The Transport Layer Application Layer HTTP, HTTPS, SMTP, POP3, IMAP, FTP, SFTP telnet, SSH, daytime, echo, DNS, Transport Layer Internet Layer TCP UDP IP Dominant protocols UDP (User Datagram Protocol) TCP (Transmission Control Protocol) 94

95 UDP UDP characteristics Connections are not persistent Connections are unreliable Adds port number to IP protocol Why UDP? Fast Not all applications need reliable transmission E.g. streaming audio E.g. DNS We will not use UDP in COS

96 TCP TCP characteristics Connections are persistent Virtual circuit Connections are reliable Reassembles packets in proper order Requests resend of missing/corrupted packets Ordered reliable byte stream Recall file descriptors Most Internet apps use TCP We will use TCP in COS

97 TCP Headers Header Source port Destination port 97

98 Ports Port A software abstraction 16-bit integer Identifies unique process on specified host Client and server communicate via ports Known port on server Ephemeral port on client Ports allow comm between specific processes on specific hosts (as opposed to comm between hosts) 98

99 Port Numbers : Well-Known Server Ports Used by common servers (HTTP, FTP, etc.) : Registered Server Ports Registered by software vendors to reduce likelihood of conflicts : Dynamic/Private Ports Available for any purpose 99

100 Well-Known/Registered Ports Some Well-Known Server Ports Port Kind of Process 7 echo 13 daytime 20 FTP-control 21 FTP-data 22 SSH, SFTP 23 telnet 25 SMTP Port Some Registered Server Ports 3306 MySQL Kind of Process 8080 Apache Tomcat 53 DNS 80 HTTP 110 POP2 143 IMAP 443 HTTPS 100

101 The Application Layer Application Layer HTTP, HTTPS, SMTP, POP3, IMAP, FTP, SFTP telnet, SSH, daytime, echo, DNS, Transport Layer Internet Layer TCP UDP IP Many protocols, each with its own header/payload format 101

102 Application Layer Protocols Some application layer protocols: Protocol daytime echo HTTP (Hypertext Transfer Protocol) HTTPS (Hypertext Transfer Protocol Secure) SMTP (Simple Mail Transfer Protocol) POP3 (Post Office Protocol 3) IMAP (Internet Message Address Protocol) FTP (File Transfer Protocol) SFTP (Secure File Transfer Protocol) Telnet SSH (Secure Shell) DNS (Domain Name System) Internet Application Time of Day Echo World Wide Web File Transfer Remote Shell DNS 102

103 Exploring App Layer Protocols Telnet program Remote shell Provides access to a command-line interface on a remote host Similar to ssh, but with no encryption Has its own (vacuous) app layer protocol and default port (23) 103

104 Exploring App Layer Protocols Telnet program Given computer (IP address or domain name) and port Establishes connection with app running on specified computer at specified port Can use to explore other app level protocols 104

105 Exploring the Daytime Protocol Daytime protocol (port 13) Try on CourseLab: telnet time-a.nist.gov

106 Exploring the Echo Protocol Echo protocol (port 7) No publicly available echo server, so I wrote one Access courselab01.cs.princeton.edu, port Try on CourseLab: telnet courselab01.cs.princeton.edu

107 Exploring the FTP Protocol On CourseLab: From ftp-using-raw-commands-and-telnet/ $ telnet ftp.ietf.org 21 Trying Connected to ftp.ietf.org. Escape character is '^]'. 220 FTP server ready USER anonymous 331 Anonymous login ok, send complete address as your password PASS your address 230 Anonymous access granted, restrictions apply CWD ietf/ftpext/ 250 CWD command successful PASV 227 Entering Passive Mode (64,170,98,33,151,31). IP address: Port: 151 * =

108 Exploring the FTP Protocol On CourseLab, in a second window: $ telnet Trying Connected to Escape character is '^]'. 108

109 Exploring the FTP Protocol Back to the first window: $ telnet ftp.ietf.org 21 Trying Connected to ftp.ietf.org. Escape character is '^]'. 220 FTP server ready USER anonymous 331 Anonymous login ok, send complete address as your password PASS your address 230 Anonymous access granted, restrictions apply CWD ietf/ftpext/ 250 CWD command successful PASV 227 Entering Passive Mode (64,170,98,33,151,31). RETR ftpext-charter.txt 150 Opening ASCII mode data connection for ftpext-charter.txt (6060 bytes) 226 Transfer complete QUIT 221 Goodbye. $ Document appears in the second window 109

110 Exploring the HTTP Protocol HTTP protocol (port 80) Try on CourseLab: telnet 80 Type: GET /index.html HTTP/1.1<Enter> Host: <Enter> 110

111 Exploring the HTTP Protocol HTTP protocol (port 80) Try on CourseLab: telnet 80 Type: GET /index.html HTTP/1.1<Enter> Host: <Enter> Yields HTTP redirect to

112 Aside: Other HTTP Clients Try on CourseLab curl wget and, of course firefox 112

113 Exploring the HTTPS Protocol HTTPS protocol (port 443) Too complex for telnet! Try on CourseLab: openssl s_client connect Type (quickly!!!) GET /index.html HTTP/1.1<Enter> Host: <Enter> 113

114 Aside: Other HTTPS Clients Try on CourseLab: curl wget and, of course firefox 114

115 Internet Protocol Summary HTTP SMTP DNS TCP UDP Provide platform for Internet app development LAN Interface 1 IP LAN Interface 2 Provides independence from LAN technologies LAN Interface 3 115

116 Summary We have covered: Network programming in Java and Python How to create a client How to create a (multithreaded) server How to serialize and communicate objects Network basics (as time allows) The Internet (as time allows) 116

117 Appendix 1: Network Programming in C 117

118 Echo in C See echoclient.c and echoserver.c Try: echoserver someport Try: echoclient localhost someport Try: echoclient localhost someport 118

119 Echo in C See echoclient.c Same as before See echoservermult.c Forks multiple processes Try: echoservermult someport Try: echoclient localhost someport Try: echoclient localhost someport 119