DISTRIBUTED SYSTEMS [COMP9243] Distributed Object based: Lecture 7: Middleware. Slide 1. Slide 3. Message-oriented: MIDDLEWARE

Transcription

1 DISTRIBUTED SYSTEMS [COMP9243] Distributed Object based: KINDS OF MIDDLEWARE Lecture 7: Middleware Objects invoke each other s methods Slide 1 ➀ Introduction ➁ Publish/Subscribe Middleware ➂ Map-Reduce Middleware ➃ Distributed Object Middleware Remote Objects & CORBA Distributed Shared Objects & Globe Slide 3 Customer Manager Bank newaccount() closeaccount() getaccount() AccountDB lookup() add() remove() Account withdraw() Account deposit() withdraw() Account getbalance() deposit() withdraw() getbalance() deposit() getbalance() Account withdraw() Account deposit() withdraw() Account getbalance() deposit() withdraw() getbalance() deposit() getbalance() Message-oriented: MIDDLEWARE Messages are sent between processes Message queues Machine A Machine B Machine C Sender A Distributed applications Application Application Slide 2 Network OS services Middleware services Network OS services Network OS services Slide 4 Receive queue Send queue Message R2 Kernel Kernel Kernel Application R1 Network Application Router Receiver B KINDS OF MIDDLEWARE 1 KINDS OF MIDDLEWARE 2

2 Coordination-based: Tuple space Transaction Processing Monitors: Slide 5 A Write A B Write B T Read T Insert a copy of A Tuple instance Insert a copy of B A B B B A C Look for tuple that matches T C Return C (and optionally remove it) Slide 7 Transaction Client application Requests Reply Reply Request Request TP monitor Reply Request Reply Server Server Server A JavaSpace Publish/Subscribe Web Services: Subscriber Subscriber Stock Servicequery_stock buy sell Client XML RPC Bank Servicebalance tansfer HTTP HTTP Slide Data Item 01 Subscription Slide 8 XML RPC Match Match Publish/Subscribe Middleware Auction Servicesearch get_auction manage_auction bid SOAP Photo Servicesearch add_photo delete_photo update_photo HTTP KINDS OF MIDDLEWARE 3 KINDS OF MIDDLEWARE 4

3 Map-Reduce: CHALLENGES Transparency: loose coupling good transparency Slide 9 Slide 11 Scalability: Potentially good due to loose coupling In practice hard to achieve Number of subscriptions Number of messages Flexibility: Loose coupling gives good flexibility Language & platform independence Policy separate from mechanism Programmability: Inherent distributed design Doesn t use non-distributed concepts EXAMPLES Slide 10 PUBLISH/SUBSCRIBE (EVENT-BASED) MIDDLEWARE Subscriber Subscriber Data Item 01 Subscription Match Match Publish/Subscribe Middleware Slide 12 Real-time Control Systems: External events (e.g. sensors) Event monitors Stock Market Monitoring: Stock updates Traders subscribed to updates Network Monitoring: Status logged by routers, servers Monitors screen for failures, intrusion attempts Enterprise Application Integration: Independent applications Produce output as events Consume events as input Decoupled CHALLENGES 5 MESSAGE FILTERING 6

4 MESSAGE FILTERING Subscriber Slide 13 Topic-based Data item: comp.os.distributed Match comp.os.unix Publish/Subscribe Middleware Subscription: comp.os.* Slide 15 COMMUNICATION Point-to-point Multicast Content-based routing Content-based Subscriber point-to-point based router network make forwarding decisions based on message content store subscription info at router nodes Data item: name=john age=30 Subscription: name=john Match name=john gender = male Publish/Subscribe Middleware ARCHITECTURE Slide 14 Centralised: Send Event Broker Send Subscribe Send Event Subscriber Peer-to-Peer: Subscriber 01 Send 01Event Subscriber Subscriber Send 01Event Subscriber Slide 16 REPLICATION Replicated Brokers: Copy subscription info on all nodes Keep nodes consistent What level of consistency is needed? Avoid sending redundant subscription update messages Multicast-based: Partitioned Brokers: Subscriber Subscriber Send Event No Match Match No Match Different subscription info on different nodes Events have to travel through all nodes Route events to nodes that contain their subscriptions COMMUNICATION 7 FAULT TOLERANCE 8

5 FAULT TOLERANCE MAP-REDUCE Reliable Communication: Reliable multicast Slide 17 Process Resilience (Broker): Process groups Active replication by subscribing to group messages Slide 19 Routing: Stabilise routing if a broker crashes Lease entries in routing tables EXAMPLE SYSTEMS TIB/Rendezvous: Topic-based Multicast-based CONTEXT Computations conceptually straightforward, but: Slide 18 Java Message Service (JMS): API for MOM Topic-based centralised or peer-to-peer implementations possible Scribe: Topic-based Peer-to-peer architecture, based on Pastry (DHT) Topics have unique IDs and map onto nodes Multicast for sending events Slide 20 Input data is usually large Need to finish in reasonable time Computations widely distributed (thousands of machines) How to: Parallelize the computation? Distribute the data? Handle failures? Balance the load? Tree is built up as nodes subscribe MAP-REDUCE 9 SOLUTION 10

6 SOLUTION EXAMPLE: WORD COUNT Slide 21 Map-Reduce: New abstraction for simple computations. Hide dirty details. Based on map and reduce primitives from Lisp (functional language). Basic computation: Takes set of input <key, value> pairs Produces set of output <key, value> pairs Implementation: Google s version: MapReduce Open source version: Hadoop Slide 23 EXAMPLE: WORD COUNT Count word occurances in in collection of documents: Slide 22 User supplied functions: Map Accepts: one input pair <key, value> Produces: a set of intermediate <key, value> pairs System groups intermediate values with same key together. Reduce Accepts: intermediate key, set of values for that key Produces: output list (typically small) More formally: map(k 1,v 1) list(k 2,v 2) reduce(k 2,list(v 2)) list(v2) Slide 24 map(string key, String value): // key: document name // value: document contents for each word w in value: EmitIntermediate(w, "1"); reduce(string key, Iterator values): // key: a word // values: a list of counts int result = 0; for each v in values: result += ParseInt(v); Emit(AsString(result)); EXAMPLE: WORD COUNT 11 EXECUTION OVERVIEW 12

7 EXECUTION OVERVIEW WORKER FAULT TOLERANCE Slide 25 Slide 27 Unreachable workers: Master pings workers periodically Unreachable workers are marked as failed. Tasks from failed workers reset to idle and rescheduled Completed map tasks need restart too (results on local disks) Completed reduce tasks not rescheduled (results on GFS) Map task first executes on A, then fails, then executed on B: Notify workers. Works well according to paper: Network upgrade disabled 80 machines at a time, but MapReduce continued to make progress. MASTER Data structures: Slide 26 State of each map task and each reduce task (idle, in-progress, completed) Identity of worker machines (for non-idle tasks) Location of intermediate file regions (propagate from map to reduce tasks) Fault tolerance: Data structures could be checkpointed to guard against failure In practice: Failure is unlikely On failure: Restart MapReduce Slide 28 Bad code: Sometimes user code crashes Ideally: Fix bug and re-run, but not always feasible Signal handler in worker catches crashes and sends last gasp packet to master, with sequence number of record If master records multiple failures on same record, the record is skipped on re-execution WORKER FAULT TOLERANCE 13 LOCALITY 14

8 CHALLENGES Slide 29 Network is scarce resource GFS divides files into blocks LOCALITY Each block is replicated (default: 3 replicas) MapReduce tries to schedule a map task on a machine that has a replica If that fails, schedule map task close to replica Result: For large MapReduce operations, significant fraction of input data is read locally. Slide 31 Transparency Failure transparency Reliability Dealing with partial failures Scalability Number of clients of an object Distance between client and object Design Must take distributed nature into account from beginning Performance Flexibility DISTRIBUTED OBJECTS OBJECT MODEL Classes and Objects Class: defines a type Object: instance of a class Slide 30 Slide 32 Interfaces Object references Active vs Passive objects Persistent vs Transient objects Static vs Dynamic method invocation CHALLENGES 15 REMOTE OBJECT ARCHITECTURAL MODEL 16

9 REMOTE OBJECT ARCHITECTURAL MODEL OBJECT SERVER Client Interface Server Object State Methods Object: State & Methods Implements a particular interface Server Slide 33 Proxy Skeleton Run Time System Run Time System Client OS Server OS Remote Objects: Single copy of object state (at single object server) All methods executed at single object server All clients access object through proxy Object s location is location of state Interface Slide 35 Skeleton: Server stub Static vs dynamic skeletons Run-Time System: Dispatches to appropriate object Invocation policies Object Server: Hosts object implementations Transient vs Persistent objects Concurrent access Support legacy code Skeleton Run Time System Server OS CLIENT Slide 34 Client Process: Binds to distributed object Invokes methods on object Proxy: Proxy: RPC stub + destination details Binding causes a proxy to be created Responsible for marshaling Static vs dynamic proxies Usually generated Client Proxy Run Time System Client OS Slide 36 OBJECT REFERENCE Local Reference: Language reference to proxy Proxy Run-Time System: Provides services (translating references, etc.) Send and receive OBJECT SERVER 17 OBJECT REFERENCE 18

10 Remote Reference: Server address + object ID OBJECT REFERENCE Object name (human friendly, object ID, etc.) Naming Service Slide 37 OR address id type Remote Object Slide 39 OR Proxy name Remote Object Proxy What are the drawbacks and/or benefits of each approach? BINDING AND NAME RESOLUTION Reference to proxy code (e.g., URL) & init data Name Resolution: Name remote reference Slide 38 OR URL to proxy code init Remote Object Slide 40 Reference info contained in name (e.g., URL), or Naming service stores name to reference mappings Binding: Remote reference local reference Create a proxy Connect proxy to object server OBJECT REFERENCE 19 REMOTE METHOD INVOCATION (RMI) 20

11 REMOTE METHOD INVOCATION (RMI) STATIC VS DYNAMIC INVOCATION Standard invocation (synchronous): Client invokes method on proxy Proxy performs RPC to object server Slide 41 Skeleton at object server invokes method on object Object server may be required to create object first Slide 43 Other invocations: Asynchronous invocations Persistent invocations Notifications and Callbacks INVOCATION SEMANTICS Local method invocation: executed exactly once Problem: Cannot make such a guarantee for remote invocations! Features: CORBA Object Management Group (OMG) Standard (version 3.1) Slide 42 Fault Tolerance Measure Retransmit Filter Re-execute, Invocation Request Duplicates or Re-reply Semantics No n/a n/a Maybe Yes No Re-execute procedure At-least-once Yes Yes Retransmit reply At-most-once Slide 44 Range of language mappings Transparency: Location & some migration transparency Invocation semantics: at-most-once semantics by default; maybe semantics can be selected Services: include support for naming, security, events, persistent storage, transactions, etc. What s the difference between Maybe and At-most-once? STATIC VS DYNAMIC INVOCATION 21 CORBA ARCHITECTURE 22

12 CORBA ARCHITECTURE Interoperable Object Reference (IOR) Can be shared between different implementations Client machine Server machine Slide 45 Static IDL proxy Client application Dynamic Invocation Interface ORB interface Object adapter Object implementation Skeleton Dynamic Skeleton Interface ORB interface Slide 47 Repository identifier Profile ID Tagged Profile Profile Interoperable Object Reference (IOR) Client ORB Local OS Server ORB Local OS IIOP version Host Port Object key Components Network POA identifier Object identifier Other serverspecific information INTERFACES: OMG IDL BINDING Example: A Simple File System: Direct Binding: Slide 46 module CorbaFS { interface File; // forward declaration interface FileSystem { exception CantOpen {string reason;}; enum OpenMode {Read, Write, ReadWrite}; File open (in string fname, in OpenMode mode) raises (CantOpen); }; interface File { // an open file string read (in long nchars); void write (in string data); void close (); }; }; Slide 48 Create proxy ORB connects to server (using info from IOR) Invocation requests are sent over connection Indirect Binding: Client IOR IOR refers to implementation repository 1. First invocation or binding request 5. Actual invocation 4. Redirect message 3. Ack object is active Implementation repository Object server 2. Activate/start object INTERFACES: OMG IDL 23 CORBA SERVICES 24

13 DISTRIBUTED SHARED OBJECT (DSO) MODEL CORBA SERVICES State Network Local Representative (Local Object) Slide 49 Some of the standardised services are the following: Naming Service Event Service Transaction Service Security Service Fault Tolerance Slide 51 Interface Process Distributed Shared Object Distributed Shared Objects: Object state can be replicated (at multiple object servers) Object state can be partitioned Methods executed at some or all replicas Object location no longer clearly defined CORBA BIBLIOGRAPHY Slide 50 [1] IIOP Complete, W. Ruh, T. Herron, and P. Klinker, Addison Wesley, [2] The Common Object Request Broker: Architecture and Specification (2.3.1), Object Management Group, [3] C Language Mapping Specification, Object Management Group, [4] CORBAservices: Common Object Services Specification, Object Management Group, Play with CORBA. Many implementations available, including ORBit: Slide 52 CLIENT Client has local representative (LR) in its address space Stateless LR Equivalent to proxy Methods executed remotely Statefull LR Full state Partial state Methods (possibly) executed locally Client app LR DISTRIBUTED SHARED OBJECT (DSO) MODEL 25 OBJECT 26

14 OBJECT GLOBE (GLOBAL OBJECT BASED ENVIRONMENT) Scalable wide-area distributed system: Wide-area scalability requires replication Wide-area scalability requires flexibility Slide Remote Object Replicated Object Slide 55 Features: Per-object replication and consistency Per-object communication Mechanism not policy Transparency (replication, migration) Dynamic replication Partitioned Object Replicated and Partitioned Object LOCAL REPRESENTATIVE OBJECT SERVER Slide 54 Server dedicated to hosting LRs Provides resources (network, disk, etc.) Static vs Dynamic LR support Transient vs Persistent LRs Security mechanisms LR Obj A LR Obj B Slide 56 Replication subobject Control subobject Same interface as implemented by semantics subobject Semantics subobject Location of LRs: Communication subobject LRs only hosted by clients Statefull LRs only hosted by object servers Statefull LRs on both clients and object servers Communication with other local objects GLOBE (GLOBAL OBJECT BASED ENVIRONMENT) 27 LOCAL REPRESENTATIVE 28

15 Semantics Subobject: Equivalent to CORBA Servant Implements object interface (Globe IDL) State stored in semantics subobject Slide 57 Control Subobject: Implements standard control interface Mediates between replication and semantics subobjects Does marshaling/unmarshaling START INVOKE SEND RETURN Slide 59 HOMEWORK How could you improve the performance of the shuffle phase of map-reduce? Hacker s edition: Implement a simple remote filesystem using a freely available version of CORBA (or other middleware if you prefer). Replication Subobject: Implements standard replication interface: start(), send(), invoked() Responsible for replication and consistency Implementation independent of semantics subobject REPLICATION EXAMPLE: ACTIVE REPLICATION Slide 58 Repl Comm Local object process B Ctrl Sem Local object process A Repl Comm Ctrl Process A has no semantics subobject Repl Comm Local object process C Ctrl Sem Slide 60 READING LIST MapReduce: Simplified Data Processing on Large Clusters Google s MapReduce Globe: A Wide-Area Distributed System An overview of Globe Request path Reply path Replication subobject forwards invocation to A and B and later merges results CORBA: Integrating Diverse Applications Within Distributed Heterogeneous Environments An overview of CORBA HOMEWORK 29 READING LIST 30