Welcome Aboard! CprE 450/550 Distributed Systems and Middleware. Instructor: Yong Guan 3219 Coover Tel: (515)

Transcription

1 CprE 450/550 Distributed Systems and Middleware Welcome Aboard! Instructor: Yong Guan 3219 Coover Tel: (515)

2 The insider s view of a Centralized OS The insider s view of a centralized OS. (Roughly patterned after XINU [Comer 1984]) user programs file system device manager and device drivers real-time clock manager process coordinator process manager memory manager hardware remote files + remote device access + remote process management

3 Natural Extensions Two Trends System Virtualization Allow multiple instances of (possibly different) OSes on a single machine Distributed OS Large scale networked systems/machines Emerging Distributed Apps Cloud Computing P2P file sharing Data outsourcing: Google Docs, Data centers, Yahoo Photo Album, etc. IPTV/VoIP Web2.0 & WebOS, Wikipedia Social websites: Myspace, Facebook, LinkedIn,.

4 New Developments in DS areas Virtualization Blockchain

5 Course Description Fundamentals of distributed computing Naming Services Synchronization and Consistency Distributed File Systems Distributed Object-based Systems System Virtulization Security Mobility Support Blockchain Emerging Apps: P2P, VoIP, New Architecture Design for the future Internet Middleware CORBA RPC Blockchain-based Middleware Middleware-based Application Design and Development

6 Course Outline and Schedule Use video-recorded lectures (Students watch them on Canvas) Meet in the classroom every other week for updated material and project assignments In-class lecture schedule will be posted on Canvas Office hours held weekly on Tuesday, 11-noon, both face-toface and online. In-class midterm exam and take-home final exam Final term paper presentation in the final exam week

7 Course Assignment Three programming projects, two of which are small, and the 3 rd is a larger one One term paper Undergraduate: 6-pages literature survey on a specific topic based on the reading of at least 10 papers published within the past five years. Graduate: 8-pages term paper, including defining a specific problem, surveying existing work, developing a (better) solution, and evaluating your results Two exams (in-class mid-term and take-home final exams) 6 online quizzes A small number of homework (no need to submit), demonstration, and presentations

8 Grading Criteria Total point is 100 Mid-term & Final Exam: 40 Course Projects: 30 Term papers: 25 Vision Related work Critiques: identify holes/questions and potential improvements of existing work Depth and breadth of knowledge to the materials Quizzes and term paper presentation: 5 Attendance for those in-class lectures is expected and encouraged. Bonus 3 points.

9 Basics of Operating Systems and Computer Networks: A Review Dr. Yong Guan Department of Electrical and Computer Engineering & Information Assurance Center Iowa State University

10 Outline for Today s Talk OS and Networking Basics

11 Required Readings for Today s Talk Required Readings: None Reference books: Andrew Tanenbaum, Modern Operating Systems, 2nd Edition, Prentice Hall, Inc., 2001, ISBN: Alberto Leon-Garcia and Indra Widjaja, Communication Networks: Fundamental Concepts and Key Architectures, 1st Edition, McGraw-Hill Companies, Inc., 2000, ISBN

12 What is an OS? Doom, XXI emacs gcc OS hardware software between applications and reality: abstracts hardware and makes portable makes finite into (near) infinite provides protection

13 Process!= Program Program: code + data passive Process: running program state: registers, stack, heap position: Program counter We both run netscape: same program, different process int c; int main() { printf( hello ); } Heap Process vs. Thread Thread = pointer to instruction + state Process = thread + address space stack data code int a; main()

14 What have we learned about OS? OS Centralized Version Important things we have discussed Multi-user, multi-process, multi-thread Synchronization, Mutual Exclusion, Deadlock Scheduling Memory I/O Devices Files, and File System

15 Extensions Two Trends System Virtualization Allow multiple instances of (possibly different) OSes on a single machine Distributed OS Large scale networked systems/machines Emerging Distributed Apps P2P file sharing Data outsourcing: Google Docs, Data centers, Yahoo Photo Album, etc. IPTV/VoIP Web2.0 & WebOS: Myspace, Wikipedia, etc.

16 Basics of Computer Networks

17 ISO OSI Reference Model Seven layers Lower three layers are peer-to-peer Next four layers are end-to-end Application Presentation Session Transport Network Datalink Physical Network Datalink Physical Physical medium Application Presentation Session Transport Network Datalink Physical

18 OSI vs. TCP/IP OSI: conceptually define services, interfaces, protocols Internet: provide a successful implementation Application Presentation Session Transport Network Datalink Physical OSI Application Transport Internet Host-tonetwork TCP/IP Telnet FTP DNS TCP LAN IP UDP Packet radio

19 Internetworking and the Internet Protocol (IP) CprE

20 IP Related Protocols Application Application TCP UDP Network Layer ICMP IP ARP RARP Physical Network

21 The Internet Network Layer Host / Router network layer functions: Transport Layer: TCP, UDP, OSFP, etc. Network Layer Routing protocols path selection RIP, OSPF, BGP Routing Table IP protocol addressing conventions datagram format packet handling conventions ICMP protocol error reporting router signaling Link Layer physical Layer

22 IPv4 Service Model IPv4 [RFC 791] Datagram-switching: Connectionless model. Best-effort delivery: Unreliable service packets can get lost packets can be delivered out of order duplicate copies of a packet are delivered packets can be delayed for a long time Datagram format 64KB maximum size IP Header (20B min) Current version IPv4 IPv6 is next generation IP All vendors support it IP datagram data payload Version HLen TOS Length Ident Flags Offset TTL Protocol Checksum SourceAddr DestinationAddr Options (variable/optional) Pad (variable) Data

23 IP Datagram Format: IP Header Version IHL Type of Service Total Length Max Size = 64KB Identification Flags Fragment Offset Time to Live Protocol Header Checksum Source IP Address Destination IP Address Options IP Header - HLen header length in 32-bit words (5 HLen 15) - TOS (Type of Service): Reliability, Latency, Throughput. Now split in Differentiated Service (DS) Field (6 bits) other two bits used by ECN (Early Congestion Notification) - Length the length of the entire datagram/segment; header + data - Flags: Don t Fragment (DF); More Fragments (MF) - Fragment Offset all fragments excepting last one contain multiples of 8 bytes - Header Checksum - uses 1 s complement (IP Checksum) Padding

24 IP Datagram Format IP protocol version number header length (words) Type of Service Time-to-Leave max number remaining hops (decremented at each router) Upper layer protocol to deliver payload to ICMP=1, TCP=6, UDP=17, OSPF=89, etc. 32 bits ver head. type of len service length fragment 16-bit identifier flgs offset time to upper Internet live layer checksum 32 bit source IP address 32 bit destination IP address Options (if any) data (variable length, typically a TCP or UDP segment) total datagram length (bytes) fragmentation / reassembly minimum fragment size = 576 Options timestamp, record route taken, specify list of routers to visit, etc.

25 Fragmentation and Reassembly Each network has some MTU Maximum Transmission Unit (MTU) maximum packet size that can be carried by a L2 link. Path MTU := min i { MTU i } IP Strategy: fragment when necessary (if MTU < Datagram); avoid fragmentation at source host re-fragmentation is possible (if MTU < Fragment) fragments are self-contained datagrams delay reassembly until destination host do not recover from lost fragments ATM: use CS-PDU (not ATM cells) Message Source Router Destination IP IP MTU-1 MTU-2 Networ k Networ k

26 Fragmentation and Reassembly Example Maximum Transmission Units DLL MTU (Bytes) 802.3/ Ethernet 1492 FDDI (4/16) 4464/17914 X PPP 296 Hyperchannel Ident = x Start of header 0 Rest of header Offset= data bytes Ident = x Start of header 1 Rest of header 512 data bytes Offset=0 Start of header H1 R1 R2 R3 H8 Ident = x 1 Rest of header 512 data bytes Offset=512 ETH IP (1400) FDDI IP (1400) PPP IP (512) PPP IP (512) PPP IP (376) ETH IP (512) ETH IP (512) ETH IP (376) Ident = x Start of header 0 Rest of header 376 data bytes Offset=1024

27 Getting a Datagram from Source to Dest IP Datagram: misc fields source IP addr dest IP addr data Datagram remains unchanged, as it travels source to destination Addr fields of interest here A routing table in A Dest. Net. next router Nhops B E

28 Getting a Datagram from Source to Dest misc fields data Starting at A, given IP datagram addressed to B: look up net. address of B find B is on same net. as A link layer will send datagram directly to B inside link-layer frame B and A are directly connected A B Dest. Net. next router Nhops E

29 Getting a Datagram from Source to Dest misc fields data Starting at A, dest. E: look up network address of E E on different network A, E not directly attached routing table: next hop router to E is link layer sends datagram to router inside link-layer frame datagram arrives at continued.. A B Dest. Net. next router Nhops E

30 Getting a Datagram from Source to Dest misc fields data Arriving at , destined for look up network address of E E on same network as router s interface router, E directly attached link layer sends datagram to inside link-layer frame via interface datagram arrives at !!! (hooray!) Dest. next network router Nhops interface A B E

31 Commands arp -a The arp cache of the host; can also map IP name IP number ifconfig -a Lists all the interfaces of a host netstat Very powerful command, with many parameters traceroute / ping trace route and report RTTs from intermediate routers

32 Transport Layer Message Stream (UDP) Byte-Stream (TCP)

33 IP Related Protocols Application Application TCP UDP Transport Layer ICMP IP ARP RARP Physical Network

34 User Datagram Protocol (UDP) Unreliable and unordered datagram service [RFC 768] No flow control Simple Packet Multiplexor for Transport Users Endpoints identified by port numbers: ports are the demultiplexing keys servers have well-known ports see /etc/services on Unix A1 App A2 App Host A A1 App UDP OS Host B UDP OS B1 App UDP B2 App IP IP

35 Simple Packet Multiplexor (UDP) Header Format [RFC 768] Source / Destination Ports: 16 bit integers of source and sink; UDP Length: 16 bit unsigned integer for entire datagram. Size 2 16 (64KB); Fragmentation by IP Optional IP checksum: pseudo-header + UDP-header + data Source Port Destination Port UDP Length UDP Checksum Data Source IP Address Destination IP Address Protocol = 17 UDP Length

36 User-Datagram Protocol (UDP) Applications may prefer using UDP: don t need reliable delivery ; do not need connection oriented service or cannot afford the connection setup overhead ; have their own special needs, such as streaming of real-time audio or video ; tftp, RIP, DNS, SNMP, RTP, etc.

37 TCP Overview [RFC793, 1122, 1323, 2018, 2581] Connection-oriented Byte-stream Application writes bytes TCP sends segments appl. reads bytes Application Process Full duplex Flow control: keep sender from overrunning receiver Congestion control: keep sender from overrunning network Application Process write() bytes (byte stream) read() bytes TCP Send Buffer Segment stream Segment Segment Segment Transmit segments TCP Receive Buffer

38 TCP Segment Format 32 bits URG: urgent data ACK: ACK # valid PSH: push data now (generally not used) RST, SYN, FIN: connection estab (setup, teardown commands) Internet checksum (as in UDP) source port # dest port # head len sequence number acknowledgement number not used UAP R S F checksum rcvr window size application data (variable length) ptr urgent data Options (variable length) counting by bytes of data (not segments!) # bytes rcvr willing to accept

39 TCP Segment Format (cont'd) Each TCP connection is identified with 4-tuple: (TCP, SrcPort, SrcIPAddr, DsrPort, DstIPAddr) TCP Sliding Window + Flow Control: ( acknowledgment, SequenceNum, AdvertisedWindow ) Data (SequenceNum) Sender Receiver Flags FIN, RESET, PUSH, URG, ACK Acknowledgment + AdvertisedWindow

40 TCP Connection Management TCP sender, receiver establish connection before exchanging data segments Initialize TCP variables: sequence numbers buffers, flow control info (e.g., RcvWindow) client : connection initiator socket(); sockaddress; connect(); server : contacted by client socket(); sockaddress; bind(); listen(); accept(); Three way handshake: Step 1: client end system sends TCP SYN control segment to server specifies initial sequence number sides exchange initialization parameters, such as MSS. Step 2: server end system receives SYN, replies with SYNACK control segment ACKs received SYN allocates buffers specifies server receiver initial sequence number.

41 3-way Handshake Connection Establishment Upon Connection Establishment, both sides exchange the Maximum Segment Sizes (MSS) which are willing to accept: If client is non-local, MSS = 536 (default) 536 = MSS only if SYN is set Host A Figure 8.22 Host B Active participant (client) Passive participant (server)

42 TCP: Initial Sequence Number The ISN should change over time [RFC 793]. Host A ISN should be viewed as a counter which increments every 4 seconds When the same initial sequence number is chosen by a host, old segments cannot be distinguished from new current ones. Host B Delayed segment with SN ( n +2 ) will be accepted

43 Data Transfer: Sequence Numbers and ACKs Sequence Numbers: byte stream number of first byte in segment s data ACKs: seq. # of next byte expected from other side cumulative ACK TCP spec doesn t say how receiver handles out-of-order segments up to implementor Example next figure: telnet interaction; User types C host ACKs receipt of echoed C Simple telnet scenario Host A Host B B ACKs receipt of C ; echoes back C time

44 TCP Connections: Graceful Close Closing a TCP connection: TCP close is symmetric client close(); socket Step 1: client end system sends TCP FIN control segment to server (client knows when input has finished Step 2: server receives FIN, replies with ACK. Closes connection, sends FIN. Note: Connections stay for 2MSL in TIME_WAIT state; MSL = 30, 60, 120 secs Deliver 150 bytes Host A Host B TIME_WAIT closed

45 TCP: Reliable Data Transfer Simplified Sender Assume: 1. one way data transfer ; 2. no flow control ; 3. no congestion control. event: data received from application above Create and Send segment wait for event event: timer timeout for segment with seq # y Retransmit segment event: ACK received, with ACK # y ACK processing

46 TCP and POSIX Socket API socket() connect() blocking connect() returns Host A (Client) Host B (Server) socket() bind() listen() accept() blocking write() read() (blocking) read returns accept returns read() blocks read returns write() read (blocks)

47 Application Layer SMTP, DNS, HTTP, etc.

48 Internet apps: application, transport protocols Application remote terminal access Web file transfer streaming multimedia remote file server Internet telephony Application layer protocol smtp [RFC 821] telnet [RFC 854] http [RFC 2068] ftp [RFC 959] proprietary (e.g. RealNetworks) NSF proprietary (e.g., Vocaltec) Underlying transport protocol TCP TCP TCP TCP TCP or UDP TCP or UDP typically UDP

49 Questions? Thanks and See you next time

50 The Web: the http protocol http: hypertext transfer protocol Web s application layer protocol client/server model client: browser that requests, receives, displays Web objects server: Web server sends objects in response to requests http1.0: RFC 1945 http1.1: RFC 2068 PC running Explorer Server running NCSA Web server Mac running Navigator

51 The http protocol: more http: TCP transport service: client initiates TCP connection (creates socket) to server, port 80 server accepts TCP connection from client http messages (application-layer protocol messages) exchanged between browser (http client) and Web server (http server) TCP connection closed http is stateless server maintains no information about past client requests aside Protocols that maintain state are complex! past history (state) must be maintained if server/client crashes, their views of state may be inconsistent, must be reconciled

52 http example Suppose user enters URL (contains text, references to 10 jpeg images) 1a. http client initiates TCP connection to http server (process) at Port 80 is default for http server. 1b. http server at host waiting for TCP connection at port 80. accepts connection, notifying client 2. http client sends http request message (containing URL) into TCP connection socket 3. http server receives request message, forms response message containing requested object (somedepartment/home.index), sends message into socket time

53 http example (cont.) 4. http server closes TCP connection. 5. http client receives response message containing html file, displays html. Parsing html file, finds 10 referenced jpeg objects time 6. Steps 1-5 repeated for each of 10 jpeg objects

54 Non-persistent, persistent connections Non-persistent http/1.0: server parses request, responds, closes TCP connection 2 RTTs to fetch object TCP connection object request/transfer each transfer suffers from TCP s initially slow sending rate many browsers open multiple parallel connections Persistent default for htp/1.1 on same TCP connection: server, parses request, responds, parses new request,.. client sends requests for all referenced objects as soon as it receives base HTML. fewer RTTs, less slow start.

55 http message format: request two types of http messages: request, response http request message: ASCII (human-readable format) request line (GET, POST, HEAD commands) Carriage return, line feed indicates end of message header lines GET /somedir/page.html HTTP/1.0 User-agent: Mozilla/4.0 Accept: text/html, image/gif,image/jpeg Accept-language:fr (extra carriage return, line feed)

56 http request message: general format

57 http message format: response status line (protocol status code status phrase) data, e.g., requested html file header lines HTTP/ OK Date: Thu, 06 Aug :00:15 GMT Server: Apache/1.3.0 (Unix) Last-Modified: Mon, 22 Jun Content-Length: 6821 Content-Type: text/html data data data data data...

58 http response status codes In first line in server->client response message. A few sample codes: 200 OK request succeeded, requested object later in this message 301 Moved Permanently requested object moved, new location specified later in this message (Location:) 400 Bad Request request message not understood by server 404 Not Found requested document not found on this server 505 HTTP Version Not Supported

59 User-server interaction: authentication Authentication : control access to server content authorization credentials: typically name, password stateless: client must present authorization in each request authorization: header line in each request if no authorization: header, server refuses access, sends WWW authenticate: header line in response client usual http request msg 401: authorization req. WWW authenticate: usual http request msg + Authorization: <cred> usual http response msg server usual http request msg + Authorization: <cred> usual http response msg time

60 Cookies: keeping state server-generated #, serverremembered #, later used for: authentication remembering user preferences, previous choices server sends cookie to client in response msg Set-cookie: client presents cookie in later requests cookie: client usual http request msg usual http response + Set-cookie: # usual http request msg cookie: # usual http response msg usual http request msg cookie: # usual http response msg server cookiespectific action cookiespectific action

61 Conditional GET: client-side caching Goal: don t send object if client has up-to-date cached version client: specify date of cached copy in http request If-modified-since: <date> server: response contains no object if cached copy is up-to-date: HTTP/ Not Modified client http request msg If-modified-since: <date> http response HTTP/ Not Modified server object not modified http request msg If-modified-since: <date> http response HTTP/ OK <data> object modified

62 Web Caches (proxy server) Goal: satisfy client request without involving origin server user sets browser: Web accesses via web cache client sends all http requests to web cache object in web cache: web cache returns object client Proxy server origin server else web cache requests object from origin server, then returns object to client client origin server

63 Web Caches (proxy server) Assume: cache is close to client (e.g., in same network) smaller response time: cache closer to client decrease traffic to distant servers link out of institutional/local ISP network often bottleneck public Internet origin servers 1.5 Mbps access link institutional network 10 Mbps LAN institutional cache

64 DNS: Domain Name System People: many identifiers: SSN, name, passport # Internet hosts, routers: IP address (32 bit) - used for addressing datagrams name, e.g., gaia.cs.umass.edu - used by humans Q: map between IP addresses and name? Domain Name System: distributed database implemented in hierarchy of many name servers application-layer protocol host, routers, name servers to communicate to resolve names (address/name translation) note: core Internet function, implemented as application-layer protocol complexity at network s edge

65 DNS name servers Why not centralize DNS? single point of failure traffic volume distant centralized database maintenance doesn t scale! no server has all name-to-ip address mappings local name servers: each ISP, company has local (default) name server host DNS query first goes to local name server authoritative name server: for a host: stores that host s IP address, name can perform name/address translation for that host s name

66 DNS: Root name servers contacted by local name server that can not resolve name root name server: contacts authoritative name server if name mapping not known gets mapping returns mapping to local name server a NSI Herndon, VA c PSInet Herndon, VA d U Maryland College Park, MD g DISA Vienna, VA h ARL Aberdeen, MD j NSI (TBD) Herndon, VA k RIPE London i NORDUnet Stockholm m WIDE Tokyo e NASA Mt View, CA f Internet Software C. Palo Alto, CA b USC-ISI Marina del Rey, CA l ICANN Marina del Rey, CA 13 root name servers worldwide

67 Simple DNS example host surf.eurecom.fr wants IP address of gaia.cs.umass.edu 1. contacts its local DNS server, dns.eurecom.fr 2. dns.eurecom.fr contacts root name server, if necessary 3. root name server contacts authoritative name server, dns.umass.edu, if necessary 2 root name server local name server dns.eurecom.fr authorititive name server dns.umass.edu 1 6 requesting host surf.eurecom.fr gaia.cs.umass.edu

68 DNS example Root name server: may not know authoritative name server may know intermediate name server: who to contact to find authoritative name server 2 root name server local name server dns.eurecom.fr 1 8 intermediate name server dns.umass.edu 4 5 requesting host surf.eurecom.fr authoritative name server dns.cs.umass.edu gaia.cs.umass.edu

69 DNS: iterated queries recursive query: puts burden of name resolution on contacted name server heavy load? iterated query: contacted server replies with name of server to contact I don t know this name, but ask this server local name server dns.eurecom.fr root name server iterated query intermediate name server dns.umass.edu 5 6 requesting host surf.eurecom.fr authoritative name server dns.cs.umass.edu gaia.cs.umass.edu

70 DNS: caching and updating records once (any) name server learns mapping, it caches mapping cache entries timeout (disappear) after some time update/notify mechanisms under design by IETF RFC

71 DNS records DNS: distributed db storing resource records (RR) RR format: (name, value, type,ttl) Type=A name is hostname value is IP address Type=NS name is domain (e.g. foo.com) value is IP address of authoritative name server for this domain Type=CNAME name is alias name for some cannonical (the real) name is really servereast.backup2.ibm.com value is cannonical name Type=MX value is name of mailserver associated with name

72 DNS protocol, messages DNS protocol : query and reply messages, both with same message format msg header identification: 16 bit # for query, reply to query uses same # flags: query or reply recursion desired recursion available reply is authoritative

73 DNS protocol, messages Name, type fields for a query RRs in reponse to query records for authoritative servers additional helpful info that may be used