Pattern-Oriented Software Architecture Concurrent & Networked Objects

Pattern-Oriented Software Architecture Concurrent & Networked Objects Tuesday, October 27, 2009 Dr. Douglas C. Schmidt schmidt@uci.edu www.cs.wustl.edu/~schmidt/posa.ppt Electrical & Computing Engineering Department The Henry Samueli School of Engineering University of California, Irvine

The Road Ahead 2,400 bits/sec to 1 Gigabits/sec 10 Megahertz to 1 Gigahertz In general, software has not Increasing software improved productivity as rapidly or as and QoS depends effectively heavily as on hardware COTS CPUs and networks have increased by 3-7 orders of magnitude in the past decade Extrapolating this trend to 2010 yields ~100 Gigahertz desktops ~100 Gigabits/sec LANs ~100 Megabits/sec wireless ~10 Terabits/sec Internet backbone These advances stem largely from standardizing hardware & software APIs and protocols, e.g.: Intel x86 & Power PC chipsets TCP/IP, ATM POSIX & JVMs CORBA ORBs & components Ada, C, C++, RT Java

Overview of Patterns and Pattern Languages Patterns Present solutions to common software problems arising within a certain context Help resolve key design forces Capture recurring structures & dynamics among software participants to facilitate reuse of successful designs Generally codify expert knowledge & best practices Pattern Languages Define a vocabulary for talking about software development problems Provide a process for the orderly resolution of these problems Help to generate & reuse software architectures www.posa.uci.edu Flexibility The Proxy ExtensibilityPattern Dependability Predictability Scalability Efficiency

Overview of Frameworks & Components Framework An integrated collection of components that collaborate to produce a reusable architecture for a family of related applications Frameworks differ from conventional class libraries: Frameworks Semi-complete applications Domain-specific Inversion of control Class Libraries Stand-alone components Domainindependent Borrow caller s thread of control Class Library Architecture Framework Architecture Frameworks faciliate reuse of successful software designs & implementations Applications inherit from and instantiate framework components

The JAWS Web Server Framework Key Sources of Variation Concurrency models e.g.,thread pool vs. thread-per request Event demultiplexing models e.g.,sync vs. async File caching models e.g.,lru vs. LFU Content delivery protocols e.g.,http 1.0+1.1, HTTP-NG, IIOP, DICOM Event Dispatcher Accepts client connection request events, receives HTTP GET requests, & coordinates JAWS s event demultiplexing strategy with its concurrency strategy. As events are processed they are dispatched to the appropriate Protocol. Protocol Performs parsing & protocol processing of HTTP request events. JAWS Protocol design allows multiple Web protocols, such as HTTP/1.0, HTTP/1.1, & HTTP- NG, to be incorporated into a Web server. To add a new protocol, developers just write a new Protocol component & configure it into the Cached Virtual Filesystem Improves Web server performance by reducing the overhead of file system accesses when processing HTTP GET requests. Various caching strategies, such as least-recently used (LRU) or leastfrequently used (LFU), can be selected according to the actual or anticipated workload & configured statically or dynamically.

Applying Patterns to Resolve Key JAWS Design Challenges Patterns help resolve the following common challenges: Encapsulating low-level OS APIs Efficiently Demuxing Asynchronous Decoupling event demultiplexing & Operations & Completions connection management from Enhancing server configurability protocol processing Transparently parameterizing Scaling up performance via threading synchronization into components Implementing a synchronized request Ensuring locks are released queue properly Minimizing server threading overhead Minimizing unnecessary locking Using asynchronous I/O effectively Synchronizing singletons correctly

Encapsulating Low-level OS Context A Web server must manage a variety of OS services, including processes, threads, Socket connections, virtual memory, & files. Most operating systems provide lowlevel APIs written in C to access these services. APIs Problem The diversity of hardware and operating systems makes it hard to build portable and robust Web server software by programming directly to low-level operating system APIs, which are tedious, error-prone, & non-portable. Application calls methods Solution Apply the Wrapper Facade design pattern to avoid accessing low-level operating system APIs directly. Wrapper Facade data method1() methodn() void method1(){ functiona(); functionb(); } void methodn(){ functiona(); } calls calls calls API FunctionA() API FunctionB() API FunctionC() Intent : Application This pattern encapsulates data & functions provided by existing non- OO APIs within more concise, robust, portable, maintainable, & cohesive OO class interfaces. method() : Wrapper Facade : APIFunctionA functiona() : APIFunctionB functionb()

Decoupling Event Demuxing and Connection Management from Protocol Processing Context A Web server can be accessed simultaneously by multiple clients, each of which has its own connection to the server. A Web server must therefore be able to demultiplex and process multiple types of indication events that can arrive from different clients concurrently. A common way to demultiplex events in a Web server is to use select(). Problem Client Developers often tightly couple a Web server s event-demultiplexing Event Dispatcher and connectionmanagement code with its HTTP protocol-handling GET Web Server code that performs select() HTTP 1.0 processing. request In such a design, the demultiplexing and connection-management Socket code cannot be reused Client as black-box HTTP components GET s request Neither by other HTTP protocols, nor by other middleware and applications, such as ORBs and image servers. Sockets Thus, changes to Client the event-demultiplexing Connect and connection-management code will affect the Web server protocol request code directly and may introduce subtle bugs. e.g., porting it to use TLI or WaitForMultipleObjects() Solution Apply the Reactor pattern and the Acceptor-Connector pattern to separate the generic event-demultiplexing and connection-management code from the web server s protocol code.

Intent The Reactor architectural pattern allows event-driven applications to demultiplex & dispatch service requests that are delivered to an application from one or more clients. 1. Initialize phase 2. Event handling phase : Main Program : Concrete Event Con. Event The Reactor Pattern Events Reactor handle_events() register_handler() remove_handler() <<uses>> handle set Synchronous Event Demuxer select () register_handler() get_handle() handle_events() handle_event() service() notifies dispatches owns Concrete Event A handle_event () get_handle() : Reactor : Synchronous Event Demultiplexer s s select() event Event handle_event () get_handle() Concrete Event B handle_event () get_handle() Observations Note inversion of control Also note how long-running event handlers can degrade the QoS since callbacks steal the reactor s thread!

The Acceptor-Connector Pattern Intent The Acceptor-Connector design pattern decouples the connection & initialization of cooperating peer services in a networked system from the processing performed by the peer services after being connected & initialized. notifies uses Dispatcher select() uses uses uses handle_events() Transport register_handler() Transport Transport remove_handler() notifies Connector owns notifies Service owns <<creates>> owns Acceptor Connector() connect() complete() handle_event () peer_stream_ open() handle_event () set_handle() peer_acceptor_ Acceptor() Accept() handle_event () Concrete Connector <<activate>> Concrete Service A Concrete Service B <<activate>> Concrete Acceptor

Acceptor Dynamics : Application : Acceptor : Dispatcher 1.Passive-mode endpoint initialize phase 2.Service handler initialize phase 3.Service processing phase open() accept() The Acceptor ensures that passive- mode transport endpoints aren t used to read/write data accidentally And vice versa for data transport endpoints Acceptor : 2 2 1 open() ACCEPT_ EVENT : Service Service register_handler() handle_events() 2 Events register_handler() handle_event() service() There is typically one Acceptor factory per-service/per-port Additional demuxing can be done at higher layers, a la CORBA

Synchronous Connector Dynamics Motivation for Synchrony If connection latency is negligible e.g., connecting with a server on the same host via a loopback device If multiple threads of control are available & it is efficient to use a thread-per-connection to connect each service handler synchronously If the services must be initialized in a fixed order & the client can t perform useful work until all connections are established. : Application : Connector : Service : Dispatcher 1.Sync connection initiation phase 2.Service handler initialize phase 3.Service processing phase Service Addr connect() get_handle() open() Service handle_event() register_handler() Events handle_events() service()

Asynchronous Connector Dynamics Motivation for Asynchrony If client is establishing connections over high latency links If client is a single-threaded applications If client is initializing many peers that can be connected in an arbitrary order. : Application : Connector : Service : Dispatcher 1.Async connection initiation phase 2.Service handler initialize phase 3.Service processing phase Service Addr connect() get_handle() complete() open() Service register_handler() Connector CONNECT EVENT handle_event() service() handle_events() register_handler() Events

Applying the Reactor and Acceptor- Connector Patterns in JAWS The Reactor architectural pattern decouples: 1.JAWS generic synchronous event demultiplexing & dispatching logic from 2.The HTTP protocol processing it performs in response to events Reactor handle_events() register_handler() remove_handler() <<uses>> handle set Synchronous Event Demuxer select () notifies dispatches owns HTTP Acceptor handle_event () get_handle() Event handle_event () get_handle() HTTP handle_event () get_handle() The Acceptor-Connector design pattern can use a Reactor as its Dispatcher in order to help decouple: 1.The connection & initialization of peer client & server HTTP services from 2.The processing activities performed by these peer services once they are connected & initialized.

Reactive Connection Management & Data Transfer in JAWS

Scaling Up Performance via Context HTTP runs over TCP, which uses flow control to ensure that senders do not produce data more rapidly than slow receivers or congested networks can buffer and process. Since achieving efficient end-to-end quality of service (QoS) is important to handle heavy Web traffic loads, a Web server must scale up efficiently as its number of clients increases. Threading Problem Processing all HTTP GET requests reactively within a single-threaded process does not scale up, because each server CPU time-slice spends much of its time blocked waiting for I/O operations to complete. Similarly, to improve QoS for all its connected clients, an entire Web server process must not block while waiting for connection flow control to abate so it can finish sending a file to a client. Solution Apply the Half-Sync/Half-Async architectural pattern to scale up server performance by processing different HTTP requests concurrently in multiple threads. This solution yields two benefits: 1. Threads can be mapped to separate CPUs to scale up server performance via multiprocessing. 2. Each thread blocks independently, which prevents one flow-controlled connection from degrading the QoS other clients receive.

The Half-Sync/Half-Async Intent The Half-Sync/Half-Async architectural pattern decouples async & sync service processing in concurrent systems, to simplify programming without unduly reducing performance. The pattern introduces two intercommunicating layers, one for async & one for sync service processing. This pattern defines two service processing layers one async and one sync along with a queueing layer that allows services to exchange messages between the two layers. The pattern allows sync services, such as HTTP protocol processing, to run concurrently, relative both to each other and to async services, such as event demultiplexing. Pattern Sync Service Layer Queueing Layer Async Service Layer : External Event Source Sync Service 1 Sync Service 2 Sync Service 3 read() <<read/write>> <<dequeue/enqueue>> notification message Queue Async Service <<read/write>> : Async Service : Queue work() message enqueue() <<read/write>> <<interrupt>> External Event Source notification read() message : Sync Service work()

Applying the Half-Sync/Half-Async Synchronous Service Layer Pattern in JAWS Worker Thread 1 Worker Thread 2 Worker Thread 3 Queueing Layer <<get>> <<put>> <<get>> Request Queue <<get>> Asynchronous Service Layer HTTP s, HTTP Acceptor Reactor <<ready to read>> Socket Event Sources JAWS uses the Half- Sync/Half-Async pattern to process HTTP GET requests synchronously from multiple clients, but concurrently in separate threads The worker thread that removes the request synchronously performs HTTP protocol processing & then transfers the file back to the client. If flow control occurs on its client connection this thread can block without degrading the QoS experienced by clients serviced by other worker threads in the pool.

Implementing a Synchronized Request Context The Half-Sync/Half-Async pattern contains a queue. The JAWS Reactor thread is a producer that inserts HTTP GET requests into the queue. Worker pool threads are consumers that remove & process queued requests. Solution Apply the Monitor Object pattern to implement a synchronized queue. Queue Worker Thread 1 This design pattern synchronizes concurrent method execution to ensure that only one method at a time runs within an object. It also allows an object s methods to cooperatively schedule their execution sequences. Problem <<get>> when <<get>> multiple threads insert and <<get>> remove Request Queue <<put>> Worker Thread 2 Worker Thread 3 A naive implementation of a request queue will incur race conditions or busy waiting requests. e.g., multiple concurrent producer and consumer HTTP threads s, can HTTP corrupt Acceptor the queue s internal state if it is not synchronized properly. Similarly, these threads will busy wait when Reactor the queue is empty or full, which wastes CPU cycles unnecessarily. Client 2.. uses Monitor Condition wait() notify() notify_all() Monitor Object sync_method1() sync_methodn() Monitor Lock acquire() release() uses

Dynamics of the Monitor Object : Client Thread1 Pattern : Client Thread2 : Monitor Object : Monitor Lock : Monitor Condition 1. Synchronized method invocation & serialization 2. Synchronized method thread suspension 3. Monitor condition notification the OS thread scheduler automatically resumes the client thread and the synchronized method sync_method1() the OS thread scheduler automatically suspends the client thread sync_method2() acquire() dowork() acquire() dowork() release() wait() the OS thread scheduler atomically releases the monitor lock notify() 4. Synchronized method thread resumption dowork() release() the OS thread scheduler atomically reacquires the monitor lock

pplying the Monitor Object Pattern in JAWS The JAWS synchronized request queue implement the queue s not-empty and not-full monitor conditions via a pair of ACE wrapper facades for POSIX-style condition variables. HTTP <<put>> uses 2 Thread Condition wait() notify() notify_all() Request Queue put() get() Thread_Mutex acquire() release() <<get>> uses Worker Thread When a worker thread attempts to dequeue an HTTP GET request from an empty queue, the request queue s get() method atomically releases the monitor lock and the worker thread suspends itself on the not-empty monitor condition. The thread remains suspended until the queue is no longer empty, which happens when an HTTP_ running in the Reactor thread inserts a request into the queue.

Minimizing Server Threading Overhead Context Socket implementations in certain multi-threaded operating systems provide a concurrent accept() optimization to accept client connection requests and improve the performance of Web servers that implement the HTTP 1.0 protocol as follows: The operating system allows a pool of threads in a Web server to call accept() on the same passive-mode socket handle. When a connection request arrives, the operating system s transport layer creates a new connected transport endpoint, encapsulates this new endpoint with a data-mode socket handle and passes the handle as the return value from accept(). The operating system then schedules one of the threads in the pool to receive this datamode handle, which it uses to communicate with its connected client. accept() accept() accept() passive-mode socket handle accept() accept()

Drawbacks with the Half-Sync/ Half-Async Architecture Problem Although Half-Sync/Half-Async threading model is more scalable than the purely reactive model it is not necessarily the most efficient design. e.g., passing a request between the Reactor thread and a worker thread incurs: Dynamic memory (de)allocation, Synchronization operations, A context switch, & CPU cache updates This overhead makes JAWS latency unnecessarily high, particularly on operating systems that support the concurrent accept() optimization. Worker Thread 1 <<get>> <<put>> Solution Reactor <<get>> Request Queue HTTP s, Worker Thread 2 HTTP Acceptor <<get>> Apply the Leader/Followers pattern to minimize server threading overhead. Worker Thread 3

ynamics in the Leader/Followers Pattern 1.Leader thread demuxing 2.Follower thread promotion 3.Event handler demuxing & event processing 4.Rejoining the thread pool Thread 1 Thread 2 thread 1 sleeps until it becomes the leader join() join() thread 2 sleeps until it becomes the leader thread 2 waits for a new event, thread 1 processes current event join() : Thread Pool handle_events() promote_ new_leader() handle_ events() : Set event handle_event() deactivate_ handle() reactivate_ handle() event handle_event() deactivate_ handle() : Concrete Event

Applying the Leader/Followers Pattern in JAWS Two options: 1.If platform supports accept() optimization then the OS implements the Leader/Followers pattern 2.Otherwise, this pattern can be implemented as a reusable framework Set handle_events() deacitivate_handle() reactivate_handle() select() Thread Pool synchronizer join() promote_new_leader() HTTP Acceptor handle_event () get_handle() uses demultiplexes Event handle_event () get_handle() HTTP handle_event () get_handle() Although Leader/Followers thread pool design is highly efficient the Half-Sync/Half-Async design may be more appropriate for certain types of servers, e.g.: The Half-Sync/Half-Async design can reorder and prioritize client requests more flexibly, because it has a synchronized request queue implemented using the Monitor Object pattern. It may be more scalable, because it queues requests in Web server virtual memory, rather than the operating system kernel.

Problem Developing software that achieves the potential efficiency & scalability of async I/O is hard due to the separation in time & space of async operation invocations and their subsequent completion events. <<uses>> The Proactor Solution Pattern Initiator Apply the Proactor architectural pattern to make efficient use of async I/O. This pattern allows event-driven applications to efficiently demultiplex & dispatch service requests triggered by the completion of async operations, thereby achieving the performance benefits of concurrency without incurring many of its liabilities. <<uses>> <<uses>> Asynchronous Operation Processor execute_async_op() <<enqueues>> <<executes>> <<invokes>> Asynchronous Operation async_op() is associated with <<demultiplexes & dispatches>> Completion handle_event() Completion Event Queue <<dequeues>> Asynchronous Event Demuxer get_completion_event() Proactor handle_events() Concrete Completion

Dynamics : Initiator : Asynchronous in : the Asynchronous Proactor : Completion Pattern Operation Operation Event Queue Processor : Proactor Completion 1. Initiate operation 2. Process operation 3. Run event loop 4. Generate & queue completion event 5. Dequeue completion event & perform completion processing Completion Completion Ev. Queue exec_async_ operation () async_operation() Result Result event handle_events() event Result Result handle_ event() Note similarities & differences with the Reactor pattern, e.g.: Both process events via callbacks However, it s generally easier to multi-thread a proactor service()

Applying the Proactor Pattern in The Proactor pattern structures the JAWS concurrent server to receive & process requests from multiple clients asynchronously. JAWS JAWS HTTP components are split into two parts: 1. Operations that execute asynchronously e.g., to accept connections & receive client HTTP GET requests 2. The corresponding completion handlers that process the async operation results e.g., to transmit a file back to a client after an async connection operation completes <<uses>> Web Server <<uses>> <<uses>> Windows NT Operating System execute_async_op() <<enqueues>> <<executes>> <<invokes>> Asynchronous Operation AcceptEx() ReadFile() WriteFile() is associated with <<demultiplexes & dispatches>> Completion handle_event() I/O Completion Port <<dequeues>> Asynchronous Event Demuxer GetQueuedCompletionStatus() Proactor handle_events() HTTP Acceptor HTTP

Proactive Connection Management & Data Transfer in JAWS