Communication Distributed Systems Santa Clara University 2016
Protocol Stack Each layer has its own protocol Can make changes at one layer without changing layers above or below Use well defined interfaces
Protocols OSI model was never popular Physical layer: Transmission of bits Data link layer: Organizes bits into frames Add checksum to frames in order to detect / correct errors
Layered Protocols Open Systems Interconnection Reference Model
Layered Protocols
Example Data link layer
Transport Protocols Transport layer Offer a reliable connection TCP protocol vs. UDP protocol There are many new proposals for the transport layer E.g. Real-time transport protocol (RTP) for realtime data transfer
Transport Protocol Client Server TCP TCP not tailored for synchronous request-reply behavior of client-server interaction Can use UDP with added reliability features Transactional TCP: Proposal that combines connection setup with sending request.
T/TCP
Higher Level Protocols Session Protocol Enhancement of transport layer: Dialog control Synchronization Not present in IP suite Presentation layer: meaning of bits Application layer: OSI intended a collection of network applications email, file transfer, terminal emulation
Middleware Protocols Middleware lives at the application layer Can contain general purpose protocols that warrant their own layer Examples for middleware services Authentication protocols Authorization protocols Distributed commit protocols
Middleware Protocols Remote procedure calls Remote object invocation Message queuing systems Communication of continuous media through streams
Network Model for networked communication
Remote Procedure Calls Birrell & Nelson (1984) Programs call procedures on other machines Remote Procedure Call (RPC) When process on A calls procedure on B, calling process on A is suspended Information is passed through the parameters from A to B and through the result from B to A
Remote Procedure Calls Conventional procedure call uses stack
Remote Procedure Calls Conventional procedure call: Call by value Call by reference Call by copy / restore Caller copies variable to stack Callee copies overwrites variable on the stack Almost like call by reference But not if same parameter occurs several times in the parameter list
Remote Procedure Calls RPC Make remote procedure call look like a local one Use stubs at client and server
Remote Procedure Calls RPC execution 1. Client procedure calls the client stub in the normal way 2. Client stub builds a message and calls the local OS 3. Client s OS sends the message to the remote OS 4. Remote OS gives message to the server stub 5. Server stub unpacks parameters and calls server 6. Server does the work and returns the result to the stub 7. Server stub packs result in a message and calls local OS 8. Server s OS sends message to client s OS 9. Client s OS gives message to the client stub 10.Stub unpacks the result and returns to the client
Remote Procedure Calls
Remote Procedure Calls Passing reference parameters How to pass pointers: Not at all (pointer to an object) Pack what is pointed to into the message as in copy / restore
Remote Procedure Calls Caller and callee need to use the same standard for formatting data A. Message on Pentium B. Message as received on SPARC C. Message on SPARC after reformatting
Remote Procedure Calls Example
Remote Procedure Calls To allow implementation of stubs for RPC Define interface Interface Definition Language
Remote Procedure Calls RPC model assumes that caller and callee only communicate through message passing Sometimes caller and callee live in the same machine IP calls to the same machine are unnecessarily complex
Doors Some OS offer RPCs for processes collocated on the same machine Doors OS needs to support doors Sever must register a door before it can be called OS returns a door identifier E.g. Solaris: each door has a file name
Doors
Remote Procedure Calls Asynchronous RPC Request reply behavior can prevent client from doing useful work as long as it does not depend on the result Asynchronous RPC is not blocking
Remote Procedure Calls (a) Traditional RPC (b)asynchronous RPC
Remote Procedure Calls If server gives a result, can combine two asynchronous RPC to yield deferred synchronous RPC
Remote Procedure Calls One way RPC: Client does not wait for an acknowledgment
DCE RPC Distributed Computing Environment Middleware system Layer of abstraction between os and distributed application Uses client server model DCE consists of (among others) distributed file service directory service security service distributed time service
DCE RPC RPC system allows accessing remote services by calling a local procedure RPC locates correct server and binds client and server software Handles message transport, fragmentation and reassembly Data conversions
DCE RPC Interface Definiton Language IDL files define types, constants, for parameter marshaling Use IDL compiler to generate header file, client stub, and server stub
DCE RPC
DCE - RPC Client to Server Binding in DCE
DCE RPC Semantic options At most once operation Idempotent
Remote Object Invocation RPC principles can be applied also to Remote Objects Sophisticated systems CORBA and DCOM Objects encapsulate data (state) and methods Methods are made available through an interface Distributed Systems: Separate between interfaces and objects
Remote Object Invocation
Remote Object Invocation When a client binds to a distributed object Implementation of the interface the proxy is loaded into the client s address space Works like a client stub for RPC Proxy: marshals method invocation into messages unmarshals reply messages Actual object sits at a server machine Skeleton server stub: Unmarshals method invocations Marshals replies
Remote Object Invocation
Remote Object Invocation Objects can be compiled Compile-time objects Define an object by a class implemented in Java/C ++, Compilation of class definition gives code to instantiate Java / C++ objects Interfaces are compiled into client-side and serverside stubs
Remote Object Invocation Compile time objects depend on a particular programming language Alternative: Explicitly instantiate objects at run-time Allows development in several programming languages But need to result in objects that can be invoked from a remote machine Common method: object adapter Wrapper around the implementation Objects are defined in terms of their interface Implementation is registered at an adapter Adapter makes interface available for remote invocations
Remote Object Invocation Persistent vs. Transient Objects
Remote Object Invocation Binding a client to an object Systems with distributed objects can provide systemwide object references Can be passed freely around Client with object reference needs to bind to the referenced object Binding places proxy into the client s address space
Remote Object Invocation Implicit binding: Client can invoke methods using a reference to the object Implementation needs to allow client to bind to the object Explicit binding Client needs to explicitly bind and obtain a pointer to a distributed object
Remote Object Invocation Code with (a) implicit binding using only global references and (b) with explicit binding with local and global references
Remote Object Invocation Possible Implementation of implicit binding Object reference gives network address of server endpoint identifying the server that manages the object endpoint = local port dynamically assigned by server s OS indication of which object
Remote Object Invocation Drawback Server crashes and endpoint changes Object references are now invalid Could use a local daemon to keep track of the endpoints But now server cannot be relocated So use a location server which turns out to not be scalable
Remote Object Invocation Full object reference can also identify network protocol etc. could have an implementation handle that contains a complete implementation of the proxy Client follows link and installs the proxy
Remote Object Invocation After binding Client issues remote method invocation - RMI Difference between RMI and RPC RMI supports systemwide object references Support of RMI specify interface with an interface definition language or: use a single language to generate stubs automatically This is static invocation interface of an object are known when client application is developed
Remote Object Invocation Dynamic Invocation: compose a method at runtime Application selects at runtime which method it will invoke invoke(object, method, input_par, output_par)
Remote Object Invocation Example: Static invocation fobject.append(integer) Dynamic invocation invoke(fobject, id(append), int) where id(append) returns an identifier for a method append
Remote Object Invocation Example: Object browser that examines objects Browser support remote object invocation Binds to a distributed object and presents the objects interface to the user At this point, user decides on a method and parameters
Remote Object Invocation Passing parameters Example: All objects are distributed objects Use object references to interact with methods References are passed by value and copied between machines For efficiency: treat remote and local objects differently If reference is remote: Copy reference: pass by reference If reference is local: Referred object is copied pass by value
Remote Object Invocation Client program on A, server program on C Client uses local and remote reference as parameters in a RMI Copy O1 to server, pass remote reference
Remote Object Invocation Invoking a method with an object reference can result in copying an object Cannot hide this there are suddenly two replicas of the object Need to make a distinction between local and distributed objects Violates transparency Makes it harder to write distributed applications
Example 1: DCE Remote Objects DCE Distributed Object Model Added as extensions to their IDL With C++ language bindings Objects are created by a server creates local C++ objects makes methods available to clients Two types of objects: Distributed dynamic object Distributed named object
Example 1: DCE Distributed Distributed dynamic object Object Model Server creates local object on behalf of a client Accessible only to that client Clients call a create RPC DCE runtime administers the new object and associates it with the client Distributed named object Created by server to be shared amongst clients Registered with a directory service Client can look up directory service and bind to the object Registration gives Unique identifier for the object Info on how to contact the objects server
Example 1: DCE Distributed Object Model Dynamic (a) vs named objects in DCE
Example 1: DCE Distributed Remote object invocation done with RPC Object Model To invoke a method, client passes to the server method identifier identifier of interface that contains the method parameters Server maintains an object table Objects can be stored on disk instead of staying in memory Server then might have to retrieve the object first
Example 1: DCE Distributed Object Model DCE have no mechanism to handle transparent object references Clients can use binding handles associated with a named object Contains identification of an interface of the object transport protocol used to communicate with server server s host address and endpoint
Example 2: Java RMI Java: Distributed objects are part of the language Tries to maintain high transparency Differences between local and remote objects are hidden (unless it become to inefficient)
Example 2: Java RMI Java only uses remote objects for distributed objects State always resides on a single machine Interface can be made available to remote processes Interfaces are implemented by using a proxy Proxy appears as a local object in the client s address space
Example 2: Java RMI Differences between local and remote objects 1.Cloning local and remote objects is different Cloning a local object results in a new object with the same state Cloning a remote object can only be done at the server Hence, proxies are not cloned Clients need to bind to the clone if they want to use it
Example 2: Java RMI Differences between local and remote objects 2.Blocking an object Objects can be constructed as monitors" by declaring a method as synchronized" If two processes call simultaneously a synchronized method, one has to be blocked Could block client process in the client stub Problem: Need to synchronize between clients on different machines (which is complicated) Could only block at the server Problem: What happens if a client fails (which results in complications) Therefore: Java RMI only block object on proxies Remote objects are not protected natively against simultaneous access from processed operating on different proxies Will have to use an explicit distributed locking technique
Example 2: Java RMI Java Remote Object Invocation Java hides most of the differences between local and remote objects Primitive or object type can be passed as a parameter to an RMI If it can be marshaled Called serializable Example: file descriptors and sockets depend on platform > cannot be serialized
Example 2: Java RMI When passing objects during an RMI Local objects are passed by value (even if they are arrays) Remote objects are passed by reference Java RMI reference to a remote object network address and endpoint of server local identifier of the object in the server s address space protocol stack used for communication between server and client
Example 2: Java RMI Remote object is build from two different classes Server class: implementation of server side code Client class: implementation of a proxy Proxies can be marshaled Can be sent to another client Serve as references to remote objects
Example 2: Java RMI Marshaling proxies: First possibility: Convert state and code into a byte stream Would lead to very large references Better alternative: Generate implementation handle Specifies which classes are needed to construct the proxy Server might need to download classes
Comparison Java RMI allows passing proxies as parameters because every client and server runs the same JVM DCE does not share platform, so cannot pass stubs DCE needs to link to a dynamically to a locally available stub But can then use this stub as a parameter in an RPC
Message oriented communication RPC and RMI can hide communication in distributed systems Need for alternative communication services Use explicit messaging
Persistence and Communication System Synchronization
Persistence and Synchronization Persistence Message is stored until it can be delivered Sender or receiver might be down while message travels between communication servers
Persistence and Synchronization
Persistence and Synchronization Transient communication Message is stored by the communication system only as long as sender and receiver execute
Persistence and Synchronization Asynchronous communication Sender continues immediately after submitting message for transmission
Persistence and Synchronization Synchronous communication Sender blocks until its message is stored in the local buffer at the receiver or actually delivered to the receiver
Persistence and Synchronization Persistent asynchronous communication Electronic mail Persistent synchronous communication Messages are stored only at the receiving host Sender is blocked until message reaches receiver s buffer (But receiver does not have to run)
Persistence and Synchronization Transient asynchronous communication UDP One-way RPC Transient synchronous communication Sender is blocked until the message is stored in a local buffer at the receiving host Sender receives ack Sender continues Asynchronous RPC
(a) persistent asynchronous (b)persistent synchronous (c) transient asynchronous (d)receipt based transient synchronous (e) delivery based transient synchronous (f) response based transient synchronous
Persistence and Synchronization Persistent communication is necessary in largescale and widely dispersed interconnected networks Persistency is useful when dealing with failure
Message Oriented Transient Communication Berkeley UNIX sockets interface
Message Oriented Transient Communication General pattern for socket based communication
Message Oriented Transient Communication Message passing interface (MPI) Standard for message passing Supports several communication models
Message Oriented Persistent Communication Important class of middleware services: Message Queuing Systems Message Oriented Middleware (MOM) support persistent asynchronous communication offer intermediate-term storage capacity for messages
Message Oriented Persistent Communication Message Queueing Model Applications communicate by inserting messages in specific queues For sender: Guarantee that message will be eventually inserted into the recipients queue
Message Oriented Persistent Communication Message Queuing System Interface
Message Oriented Persistent Communication Message queuing system Many allow installing a handler as a callback function Invoked when message is put into the queue Allow daemon etc to monitor queue for incoming messages
Message Oriented Persistent Communication Message-queueing system architecture Local: on the same machine or the same LAN Queues distributed over network
Message Oriented Persistent Communication Message Queueing System Architecture Queues managed by queue manager Some act as routers / relays Generate an overlay network Existence simplifies network administration Only routers need to be updated when a queue is moved, inserted, or deleted Can also be used to log messages for security, fault tolerance
Message Oriented Persistent Communication
Message Oriented Persistent Communication Message Brokers Integrate existing and new applications Applications need to understand messages Sender can format outgoing messages according to the receiver or: Use a common message format or: Message brokers convert incoming messages to a format understood by receiver
Message Oriented Persistent Communication
Message Oriented Persistent Communication Message queuing systems are almost like email servers But: email servers directly support endusers have additional requirements such as automatic message filtering,
Message Oriented Persistent Communication IBM MQSeries
Message Oriented Persistent Communication IBM MQSeries Queues are managed by queue managers Message channels connect two queue managers unidirectional reliable managed by Message Channel Agent (MCA) Queue managers can be linked into the same process as an application Queues are hidden to application Standard interface for messaging
Message Oriented Persistent Communication IBM MQSeries Message channels one send queue one receive queue both MCA need to be up for sending a message Initialization: Manually or: application starts MCA or: Send queue triggers when message is put into queue, which starts a handler, to start the sending MCA or: start MCA over the network Termination: Automatic after a specific time without sending messages
Message Oriented Persistent Communication Attributes for message channel agents
Message Oriented Persistent Communication Routing tables and aliases for MQSeries
Message Oriented Persistent Communication
Stream Oriented Communication Streams: Timing is crucial Continuous media Meaning of message depends on temporal relationship to previous messages Motion, Audio Discrete media text, still images
Stream Oriented Communication Data stream: sequence of data units Asynchronous transmission mode: Data items are transmitted in order without timing constraints Synchronous transmission mode: Data items are transmitted in order Maximum end-to-end delay for each unit in the stream Isochronous transmission mode: Data units are transferred on time Maximum and minimum end-to-end delay Bounded delay jitter"
Stream Oriented Communication Simple streams Complex streams consist of several related simple streams, the sub streams
Stream Oriented Communication (a) stream between two processes (b) stream between two devices
Stream Oriented Communication Multicasting Receivers can have different requirements Use filters to adjust quality
Stream Oriented Communication Quality of Service (QoS) Flow specification: bandwidth requirements, transmission rates, delays,
Stream Oriented Communication Token bucket algorithm Generate tokens at constant rate If bucket is full, tokens fall away Each time application sends data, needs to remove tokens from bucket
Stream Oriented Communication Currently, no model for specifying QoS parameters describing resources in a communication system translating QoS parameters to resource usage
Stream Oriented Communication QoS protocol: Resource reservation Protocol (RSVP)
Stream Oriented Communication RSVP Senders provide flow specification Hand it over to RSVP process RSVP process stores specification Sender sets up path to receiver(s) providing flow specification to all intermediate nodes RSVP server when receiving a reservation request: Checks whether enough resources are available Checks whether receiver has permission to make the reservation
Stream Oriented Communication Stream synchronization Sub streams in a complex stream need to be synchronized Simple form: discrete data stream (slides) and continuous data stream (audio) Complex form: Synchronizing video and audio stream, two audio streams for stereo (with max. jitter of less than 20 μsec Need to synchronize between data units
Stream Oriented Communication Explicit synchronization at the data level
Stream Oriented Communication Synchronization at high level
Stream Oriented Communication Synchronization at high level: Multimedia middleware offers interfaces for controlling video and audio streams Multiplex different streams into a single stream: MPEG streams