Distributed Systems. Chapter 02

Transcription

1 Distributed Systems Principles and Paradigms Chapter 02 (version 31st August 2001) Maarten van Steen Vrije Universiteit Amsterdam, Faculty of Science Dept. Mathematics and Computer Science Room R4.20. Tel: (020) URL: steen 01 Introduction 02 Communication 03 Processes 04 Naming 05 Synchronization 06 Consistency and Replication 07 Fault Tolerance 08 Security 09 Distributed Object-Based Systems 10 Distributed File Systems 11 Distributed Document-Based Systems 12 Distributed Coordination-Based Systems 00 1 /

2 Layered Protocols Low-level layers Transport layer Application layer Middleware layer 02 1 Communication/2.1 Layered Protocols

3 Basic Networking Model Application Presentation Session Transport Network Data link Physical Application protocol Presentation protocol Session protocol Transport protocol Network protocol Data link protocol Physical protocol Drawbacks: Network Focus on message-passing only Often unneeded or unwanted functionality Question: Violates transparency? 02 2 Communication/2.1 Layered Protocols

4 Low-level layers Physical layer: contains the specification and implementation of bits, and their transmission between sender and receiver Data link layer: prescribes the transmission of a series of bits into a frame to allow for error and flow control Network layer: describes how packets in a network of computers are to be routed. Observation: for many distributed systems, the lowestlevel interface is that of the network layer Communication/2.1 Layered Protocols

5 Transport Layer Important: The transport layer provides the actual communication facilities for most distributed systems. Standard Internet protocols: TCP: connection-oriented, reliable, stream-oriented communication UDP: unreliable (best-effort) datagram communication Note: IP multicasting is generally considered a standard available service Communication/2.1 Layered Protocols

6 Client Server TCP TCP for transactions (T/TCP): A transport protocol aimed to support client server interaction Client Server Client Server 1 1 SYN SYN,request,FIN SYN,ACK(SYN) 2 SYN,ACK(FIN),answer,FIN ACK(SYN) request 3 ACK(FIN) FIN ACK(req+FIN) answer FIN Time 9 ACK(FIN) Time (a) (b) 02 5 Communication/2.1 Layered Protocols

7 Application Layer Observation: Many application protocols are directly implemented on top of transport protocols, doing a lot of application-independent work. News FTP WWW Transfer NNTP FTP HTTP Encoding 7-bit + MIME 7-bit text + 8-bit binary 8-bit + content type (user has to guess) Naming Newsgroup Host + path URL Distribution Push Pull Pull Replication Flooding Caching + Caching + DNS tricks Security None (PGP) Username + Password DNS tricks Username + Password 02 6 Communication/2.1 Layered Protocols

8 Middleware Layer Observation: Middleware is invented to provide common services and protocols that can be used by many different applications: A rich set of communication protocols, but which allow different applications to communicate Marshaling and unmarshaling of data, necessary for integrated systems Naming protocols, so that different applications can easily share resources Security protocols, to allow different applications to communicate in a secure way Scaling mechanisms, such as support for replication and caching Note: what remains are truly application-specific protocols Question: Such as...? 02 7 Communication/2.1 Layered Protocols

9 Remote Procedure Call (RPC) Basic RPC operation Parameter passing Variations 02 8 Communication/2.2 Remote Procedure Call

10 Basic RPC Operation Observations: Application developers are familiar with simple procedure model Well-engineered procedures operate in isolation (black box) There is no fundamental reason not to execute procedures on separate machine Conclusion: communication between caller & callee can be hidden by using procedure-call mechanism. Client Wait for result Call remote procedure Return from call Server Request Reply Call local procedure and return results Time 02 9 Communication/2.2 Remote Procedure Call

11 RPC Implementation (1/2) Local procedure call: (! #"%$ ) 1: Push parameter values of the procedure on a stack 2: Call procedure 3: Use stack for local variables 4: Pop results (in parameters) Stack pointer Main program's local variables Main program's local variables bytes buf fd return address read's local variables (a) (b) Principle: communication with local procedure is handled by copying data to/from the stack (with a few exceptions) Communication/2.2 Remote Procedure Call

12 RPC Implementation (2/2) Client machine Server machine Client process k = add(i,j) proc: "add" int: val(i) int: val(j) 1. Client call to procedure Server stub Client stub 2. Stub builds message Server process Implementation of add k = add(i,j) proc: "add" int: val(i) int: val(j) 6. Stub makes local call to "add" 5. Stub unpacks message Client OS proc: "add" int: val(i) int: val(j) Server OS 4. Server OS hands message to server stub 3. Message is sent across the network Communication/2.2 Remote Procedure Call

13 RPC: Parameter Passing (1/2) Parameter marshaling: There s more than just wrapping parameters into a message: Client and server machines may have different data representations (think of byte ordering) Wrapping a parameter means transforming a value into a sequence of bytes Client and server have to agree on the same encoding: How are basic data values represented (integers, floats, characters) How are complex data values represented (arrays, unions) Client and server need to properly interpret messages, transforming them into machine-dependent representations Communication/2.2 Remote Procedure Call

14 RPC: Parameter Passing (2/2) RPC parameter passing: RPC assumes copy in/copy out semantics: while procedure is executed, nothing can be assumed about parameter values (only Ada supports this model). RPC assumes all data that is to be operated on is passed by parameters. Excludes passing references to (global) data. Conclusion: full access transparency cannot be realized. Observation: If we introduce a remote reference mechanism, access transparency can be enhanced: Remote reference offers unified access to remote data Remote references can be passed as parameter in RPCs Communication/2.2 Remote Procedure Call

15 Local RPCs: Doors Essence: Try to use the RPC mechanism as the only mechanism for interprocess communication (IPC). Doors are RPCs implemented for processes on the same machine. Computer Client process main() {... fd = open(door_name,... ); door_call(fd,... );... } Register door Server process server_door(...) {... door_return(...); } main() {... fd = door_create(...); fattach(fd, door_name,... );... } Operating system Invoke registered door at other process Return to calling process Communication/2.2 Remote Procedure Call

16 Asynchronous RPCs Essence: Try to get rid of the strict request-reply behavior, but let the client continue without waiting for an answer from the server. Client Wait for result Client Wait for acceptance Call remote procedure Return from call Call remote procedure Return from call Request Reply Request Accept request Server Call local procedure and return results Time Server Call local procedure Time (a) (b) Variation: deferred synchronous RPC: Client Wait for acceptance Interrupt client Server Call remote procedure Request Return from call Accept request Call local procedure Return results Acknowledge Time Call client with one-way RPC Communication/2.2 Remote Procedure Call

17 RPC in Practice Essence: Let the developer concentrate on only the client- and server-specific code; let the RPC system (generators and libraries) do the rest. Uuidgen Interface definition file IDL compiler Client code Client stub Header Server stub Server code #include #include C compiler C compiler C compiler C compiler Client object file Client stub object file Server stub object file Server object file Linker Runtime library Runtime library Linker Client binary Server binary Communication/2.2 Remote Procedure Call

18 Client-to-Server Binding (DCE) Issues: (1) Client must locate server machine, and (2) locate the server. Example: DCE uses a separate daemon for each server machine. Directory machine Client machine 3. Look up server Directory server 2. Register service Server machine Client 5. Do RPC Server 1. Register endpoint 4. Ask for endpoint DCE daemon Endpoint table Communication/2.2 Remote Procedure Call

19 Remote Object Invocation Distributed objects Remote method invocation Parameter passing Communication/2.3 Remote Object Invocation

20 Remote Distributed Objects (1/2) Data and operations encapsulated in an object Operations are implemented as methods, and are accessible through interfaces Object offers only its interface to clients Object server is responsible for a collection of objects Client stub (proxy) implements interface Server skeleton handles (un)marshaling and object invocation Client machine Server machine Client invokes a method Client Proxy Client OS Same interface as object Skeleton invokes same method at object Server Skeleton Server OS Object State Method Interface Network Marshalled invocation is passed across network Communication/2.3 Remote Object Invocation

21 Remote Distributed Objects (2/2) Compile-time objects: Language-level objects, from which proxy and skeletons are automatically generated. Runtime objects: Can be implemented in any language, but require use of an object adapter that makes the implementation appear as an object. Transient objects: live only by virtue of a server: if the server exits, so will the object. Persistent objects: live independently from a server: if a server exits, the object s state and code remain (passively) on disk Communication/2.3 Remote Object Invocation

22 Client-to-Object Binding (1/2) Object reference: Having an object reference allows a client to bind to an object: Reference denotes server, object, and communication protocol Client loads associated stub code Stub is instantiated and initialized for specific object Two ways of binding: Implicit: Invoke methods directly on the referenced object Explicit: Client must first explicitly bind to object before invoking it Communication/2.3 Remote Object Invocation

23 Client-to-Object Binding (2/2)!" $# %& ' () *+ -,/.0 12 Implicit :; <= >!" <=? + -,/%@0$ :1 Explicit <= A# %& ' (B *+ -,/.012 Some remarks: Reference may contain a URL pointing to an implementation file (Server,object) pair is enough to locate target object We need only a standard protocol for loading and instantiating code Observation: Remote-object references allows us to pass references as parameters. This was difficult with ordinary RPCs Communication/2.3 Remote Object Invocation

24 Remote Method Invocation Basics: (Assume client stub and server skeleton are in place) Client invokes method at stub Stub marshals request and sends it to server Server ensures referenced object is active: Create separate process to hold object Load the object into server process... Request is unmarshaled by object s skeleton, and referenced method is invoked If request contained an object reference, invocation is applied recursively (i.e., server acts as client) Result is marshaled and passed back to client Client stub unmarshals reply and passes result to client application Communication/2.3 Remote Object Invocation

25 RMI: Parameter Passing (1/2) Object reference: Much easier than in the case of RPC: Server can simply bind to referenced object, and invoke methods Unbind when referenced object is no longer needed Object-by-value: A client may also pass a complete object as parameter value: An object has to be marshaled: Marshall its state Marshall its methods, or give a reference to where an implementation can be found Server unmarshals object. Note that we have now created a copy of the original object. Object-by-value passing tends to introduce nasty problems Communication/2.3 Remote Object Invocation

26 RMI: Parameter Passing (2/2) Machine A Machine B Local reference L1 Local object O1 Remote reference R1 Remote object O2 Client code with RMI to server at C (proxy) New local reference Copy of O1 Remote invocation with L1 and R1 as parameters Machine C Copy of R1 to O2 Server code (method implementation) Question: What s an alternative implementation for a remote-object reference? Communication/2.3 Remote Object Invocation

27 Message-Oriented Communication Synchronous versus asynchronous communications Message-Queuing System Message Brokers Example: IBM MQSeries Communication/2.4 Message-Oriented Communication

28 Synchronous Communication Some observations: Client/Server computing is generally based on a model of synchronous communication: Client and server have to be active at the time of communication Client issues request and blocks until it receives reply Server essentially waits only for incoming requests, and subsequently processes them Drawbacks synchronous communication: Client cannot do any other work while waiting for reply Failures have to be dealt with immediately (the client is waiting) In many cases the model is simply not appropriate (mail, news) Communication/2.4 Message-Oriented Communication

29 Asynchronous Communication: Messaging Message-oriented middleware: Aims at high-level asynchronous communication: Processes send each other messages, which are queued Sender need not wait for immediate reply, but can do other things Middleware often ensures fault tolerance Messaging interface Sending host Communication server Communication server Receiving host Application Routing program Buffer independent of communicating hosts Routing program Application To other (remote) communication server OS OS OS OS Local buffer Local network Internetwork Incoming message Local buffer Communication/2.4 Message-Oriented Communication

30 Persistent vs. Transient Communication Persistent communication: A message is stored at a communication server as long as it takes to deliver it at the receiver. Transient communication: A message is discarded by a communication server as soon as it cannot be delivered at the next server, or at the receiver Communication/2.4 Message-Oriented Communication

31 Messaging Combinations A sends message and continues A stopped running A sends message and waits until accepted A stopped running A B B is not running (a) Time B starts and receives message A Message is stored at B's location for later delivery B B is not running Accepted (b) Time B starts and receives message A sends message and continues Send request and wait until received A B Message can be sent only if B is running B receives message Time A Request is received B Running, but doing something else ACK Process request Time (c) (d) Send request and wait until accepted A A Send request and wait for reply Request is received B Running, but doing something else Accepted Process request Time Request is received B Running, but doing something else Process request Accepted Time (e) (f) Communication/2.4 Message-Oriented Communication

32 Message-Oriented Middleware Essence: Asynchronous persistent communication through support of middleware-level queues. Queues correspond to buffers at communication servers. Canonical example: IBM MQSeries Communication/2.4 Message-Oriented Communication

33 Basic concepts: IBM MQSeries (1/3) Application-specific messages are put into, and removed from queues Queues always reside under the regime of a queue manager Processes can put messages only in local queues, or through an RPC mechanism Message transfer: Messages are transferred between queues Message transfer between queues at different processes, requires a channel At each endpoint of channel is a message channel agent Message channel agents are responsible for: Setting up channels using lower-level network communication facilities (e.g., TCP/IP) (Un)wrapping messages from/in transport-level packets Sending/receiving packets Communication/2.4 Message-Oriented Communication

34 IBM MQSeries (2/3) Sending client Routing table Send queue Client's receive queue Receiving client Program Queue manager Queue manager Program MQ Interface Stub Server stub MCA MCA MCA MCA Server stub Stub RPC (synchronous) Local network Message passing (asynchronous) Internetwork To other remote queue managers Channels are inherently unidirectional MQSeries provides mechanisms to automatically start MCAs when messages arrive, or to have a receiver set up a channel Any network of queue managers can be created; routes are set up manually (system administration) Communication/2.4 Message-Oriented Communication

35 IBM MQSeries (3/3) Routing: By using logical names, in combination with name resolution to local queues, it is possible to put a message in a remote queue Alias table LA1 QMC LA2 QMD Routing table QMB QMC QMD SQ1 SQ1 SQ2 Alias table LA1 QMA LA2 QMD Routing table QMA SQ1 QMC SQ1 QMD SQ1 QMA SQ2 SQ1 SQ1 QMB Routing table QMA SQ1 QMC SQ2 QMB SQ1 QMD SQ1 SQ2 Alias table LA1 QMA LA2 QMC QMC SQ1 Routing table QMA SQ1 QMB SQ1 QMD SQ1 Question: What s a major problem here? Communication/2.4 Message-Oriented Communication

36 Message Broker Observation: Message queuing systems assume a common messaging protocol: all applications agree on message format (i.e., structure and data representation) Message broker: Centralized component that takes care of application heterogeneity in a message-queuing system: Transforms incoming messages to target format, possibly using intermediate representation May provide subject-based routing capabilities Acts very much like an application gateway Source client Message broker Database with conversion rules Destination client Broker program OS Queuing layer OS OS Network Communication/2.4 Message-Oriented Communication

37 Stream-Oriented Communication Support for continuous media Streams in distributed systems Stream management Communication/2.5 Stream-Oriented Communication

38 Continuous Media Observation: All communication facilities discussed so far are essentially based on a discrete, that is timeindependent exchange of information Continuous media: Characterized by the fact that values are time dependent: Audio Video Animations Sensor data (temperature, pressure, etc.) Transmission modes: Different timing guarantees with respect to data transfer: Asynchronous: no restrictions with respect to when data is to be delivered Synchronous: define a maximum end-to-end delay for individual data packets Isochronous: define a maximum and minimum end-to-end delay (jitter is bounded) Communication/2.5 Stream-Oriented Communication

39 Stream (1/2) Definition: A (continuous) data stream is a connectionoriented communication facility that supports isochronous data transmission Some common stream characteristics: Streams are unidirectional There is generally a single source, and one or more sinks Often, either the sink and/or source is a wrapper around hardware (e.g., camera, CD device, TV monitor, dedicated storage) Stream types: Simple: consists of a single flow of data, e.g., audio or video Complex: multiple data flows, e.g., stereo audio or combination audio/video Communication/2.5 Stream-Oriented Communication

40 Stream (2/2) Issue: Streams can be set up between two processes at different machines, or directly between two different devices. Combinations are possible as well. Sending process Receiving process Program OS Stream OS Network (a) Camera Display OS Stream OS Network (b) Communication/2.5 Stream-Oriented Communication

41 Streams and QoS Essence: Streams are all about timely delivery of data. How do you specify this Quality of Service (QoS)? Make distinction between specification and implementation of QoS. Flow specification: Use a token-bucket model and express QoS in that model Application Irregular stream of data units One token is added to the bucket every T Regular stream Input characteristics Maximum data unit size (bytes) Token bucket rate (bytes/sec) Token bucket size (bytes) Max. transmission rate (bytes/sec) Required Service Loss sensitivity (bytes) Loss interval (µsec) Burst loss sensitivity (data units) Min. delay noticed (µsec) Max. delay variation (µsec) Quality of guarantee Communication/2.5 Stream-Oriented Communication

42 Implementing QoS Problem: QoS specifications translate to resource reservations in underlying communication system. There is no standard way of (1) QoS specs, (2) describing resources, (3) mapping specs to reservations. Approach: Use Resource reservation Protocol (RSVP) as first attempt. RSVP is a transport-level protocol. Sender process RSVP-enabled host Application data stream Application Policy control RSVP process RSVP program Local OS Data link layer Admission control Reservation requests from other RSVP hosts Data link layer data stream Internetwork Setup information to other RSVP hosts Local network Communication/2.5 Stream-Oriented Communication

43 Stream Synchronization Problem: Given a complex stream, how do you keep the different substreams in synch? Example: Think of playing out two channels, that together form stereo sound. Difference should be less than µsec! Multimedia control is part of middleware Receiver's machine Application Application tells middleware what to do with incoming streams Middleware layer Incoming stream OS Network Alternative: multiplex all substreams into a single stream, and demultiplex at the receiver. Synchronization is handled at multiplexing/demultiplexing point (MPEG) Communication/2.5 Stream-Oriented Communication