Thread-based vs Event-based Implementation of a Group Communication Service

Transcription

1 Thread-based vs Event-based Implementation o a Group Communication Service Shivakant Mishra and Ronuan Yan Department o Computer Science University o Wyomin, P.O. Box 3682 Laramie, WY , USA. mishrajchon@cs.uwyo.edu Abstract We evaluate two techniques to implement concurrent events in a roup communication service. This evaluation is based on a comparison o the perormance measured rom two dierent implementations o a roup communication service, and our experience in usin the two techniques to implement roup communication services in the past. Our conclusion is that an event-based implementation is preerable to a thread-based implementation o a roup communication service. 1. Introduction Most roup communication services are event driven, i.e. roup members invoke various unctions on the occurrence o an event. In eneral, several events may occur concurrently at a roup member implementin a roup communication service. There are two common techniques to implement concurrent, event-driven sotware: thread-based and event-based. In the thread-based technique, a separate thread is spawned or each event type, say e type, in the proram. This thread waits or an event o type e type to occur and takes appropriate actions on the occurrence o that event. In the event-based technique, a sinle-threaded event loop perorms event demultiplexin and event handler dispatchin in response to the occurrence o multiple events. While both techniques have been used to handle concurrent events in roup communication services [2, 7, 3, 8, 1], it is not clear which o them is more suitable. The oal o this paper is to evaluate these two techniques in implementin roup communication services in terms o their eect on service perormance, complexity o their implementation, and portability. This evaluation is based on an implementation o two versions (event-based and thread-based) o the Timewheel roup communication service [5, 6], and our experience in usin these techniques to implement roup communication services in the past. The main conclusion is that an event-based technique is almost always preerable to a thread-based technique to handle concurrent events in roup communication services. An event-based implementation results in better perormance, simpler, more manaeable, and portable code. 2. Concurrent Events Events in a roup communication service occur due to interactions between a client and a roup member, interactions amon roup members, and various timeouts set by roup members. The Timewheel roup communication service [5, 6] consists o a clock synchronization protocol, a roup membership protocol, and an atomic broadcast protocol. It supports nine roup communication semantics: three kinds o orderin semantics no order, total ordered, and time ordered, three kinds o atomicity semantics weak, stron, andstrict, and one termination semantic. There are two types o interaction between a client and a roup member in the Timewheel roup communication service. A client can request to multicast a service state update with a iven pair o atomicity and order semantics, or it can request to retrieve some inormation about the replicated service state. There are two types o messaes sent by roup members to implement the Timewheel atomic broadcast protocol: a proposal messae to disseminate a service state update and a decision messae to disseminate ordinal inormation. There are three types o messaes exchaned amon members to implement the Timewheel membership protocol: a no-decision messae to inorm a ailure suspicion, a join messae to request joinin a roup, and a reconiuration messae to disseminate inormation about the list o live processes. Finally, the clock synchronization protocol uses a messae to synchronize clocks o dierent members.

2 The Timewheel roup communication service uses our types o timeouts: a decision messae timeout used to set a time by which the next decision messae is expected, a decider timeout used by a member(decider) to set a time by which the next decision messae is to be sent, a decider election timeout used by the membership protocol to elect a new decider, and a clock synchronization timeout used by the clock synchronization protocol to set a time when the next clock synchronization messae is to be sent. 3. Thread-Based Implementation The thread-based implementation o the Timewheel roup communication service uses the SGI s POSIXcompliant, pthreads library. Six threads are used in this implementation. One thread executes in a loop waitin to receive a request rom a client and, on receivin a request, invokes a unction to service that request. Another thread executes in a loop waitin to receive a messae rom the other members and, on receivin a messae, invokes a unction to process that messae. The remainin threads implement timers: one each in the atomic broadcast and clock synchronization protocols and two in the roup membership protocol. A timer is implemented as ollows: void timer(pthread mutex t *m, pthread cond t *c, struct timespec *interval) pthread mutex lock(m); pthread cond timedwait(c, m, interval); pthread mutex unlock(m); The reason or usin pthread cond timedwait unction, instead o a simple timed-wait unction to implement timers is that, in some cases, we need to unblock the timer thread beore the speciied time interval has passed. This can be done by pthread cond sinal unction call. Some o these timer threads are not always needed. For example, the second timer thread in the membership protocol is needed only when a ailure occurs and it is determined that the decider role has been lost. In our implementation, these timer threads are blocked on a pthread cond wait unction call, when they are not needed. They are unblocked by the pthread cond sinal unction call, when needed. Proress o the Timewheel roup communication service is dependent on the individual clocks o dierent processors bein synchronized. A roup member whose clock is not synchronized with the other members leaves the roup (See [4] or details). So, to ensure proress, the timer thread used by the clock synchronization protocol needs to be scheduled as soon as its timer expires. We ive this thread the hihest priority. The other ive threads are o equal priorities. A consequence o unequal priorities is that race conditions can occur between them, even when the service is implemented on a network o uni-processor systems 1. A runnin thread may et interrupted in the middle o executin a unction, and since all six threads execute in the same address space, the threads scheduled later may interere with the preempted thread. Unortunately, there is very little parallelism in roup communication services. In eneral, execution o a unction invoked by a thread must complete beore another thread can invoke a unction. So, to handle race conditions, we simply enclose most unctions by semaphores. To minimize the possibility o preemption o a runnin thread by the scheduler, we have used the io schedulin policy. To ensure that each o the ive equal-priority threads et a air share o the CPU, a runnin thread yields its control o CPU, whenever it inishes the execution o the unction invoked. 4. Event-Based Implementation There are six types o events in the event-based implementation o the Timewheel roup communication service. They correspond to the six threads in the thread-based implementation. An outline o the event loop and the processin o timeout events is shown below: void mainloop() or (;;) rset = readds; i (to ev queue!queue size() > ) to ev queue!et timeout(to); else to = NULL; ret = select(maxfd,&rset, NULL, NULL, to); i (ret == ) process to(); else or (int i = minfd; i < maxf D; i ++) i (FD ISSET(i,&rset)) ln = recvrom(i, sm, mlen,,null,&ret); event table[i]!new event(); void process to() int done =; to ev queue!irst!new event(); to ev queue!dequeue(); do 1 I the service is implemented on a network o multiprocessor systems, race conditions between threads can occur even when all threads are o equal priorities. This can happen i the thread packae can schedule more than one thread concurrently at dierent processors.

3 i (time over(clk sync to!time)) to ev queue!remove event(clk sync to); clk sync to!event(); else i (time over(to ev queue!irst!time)) to ev queue!irst!new event(); to ev queue!dequeue(); else done =1; while(!done); The event loop is implemented in a standard way usin the Unix select() system call: the receive events are reistered in the d set readds usin the FD SET unction call and the earliest timeout event is included in the timeval struct to. A queue to ev queue o timeout events is maintained that stores all timeout events reistered. This queue is sorted by the time the timeout events are suppose to occur. Because timely processin o timeout events is important, a timeout event is processed beore a messaereceipt event. To ensure that a clock synchronization timeout event is iven priority over any other timeout event, a separate variable clk sync to is maintained to record when the next clock synchronization timeout event will occur. Ater processin the irst timeout event, i the clock synchronization timeout event has occurred, it is processed. 5. Evaluation 5.1. Complexity o Implementation Concurrent events raise two related prorammin problems that need to be addressed: race conditions and schedulin. Concurrent events can cause race conditions in a thread-based implementation, and so, appropriate synchronization primitives have to be used. This adds complexity in the code. In eneral, concurrent prorammin is hard and extra attention has to be paid to ensure correctness. Because not all threads have equal priority in a threadbased implementation, a runnin thread may be preempted in the middle o a unction execution. To ensure that all threads et a air share o the CPU, they have to explicitly yield the CPU control at some loical points in their execution. Hence, the schedulin o the threads has to be controlled, to some extent, by implementors in a thread-based implementation. Since, there is only one thread o control in an eventbased implementation, the problems o race conditions or schedulin do not arise. However, in an event-based implementation, an event loop has to be implemented that schedules the processin o events in an appropriate order. Special care has to be taken in the implementation o this event loop to ensure that hiher priority events, such as the timeout event in the clock synchronization protocol, are processed irst. We ound that an event-based implementation resulted in a simpler and more manaeable code. Usin synchronization primitives correctly and controllin the schedulin o threads in the thread-based implementation was much more complicated than implementin the event loop in the eventbased implementation Portability A thread-based implementation needs a threads packae, either implemented inside the operatin system kernel or at the user level, while an event-based implementation needs a system call such as select() to examine an I/O descriptor set or event schedulin. Our experience in implementin roup communication services has been mostly on a Unix platorm. While the select() system call is portable amon all types o Unix operatin system, a threads packae is not always portable. For example, we implemented the Pinwheel atomic broadcast protocols on a network o Sun IPX SPARCstations runnin SunOS Unix operatin system [3]. This implementation used the Sun s LWP thread manaement library that runs on SunOS Unortunately, this thread manaement library does not run on other Unix operatin systems such as Solaris or IRIX. As a result, we had to do a lot o work to run the Pinwheel protocols on an IRIX operatin system platorm. So, while an event-based implementation is airly portable, the portability o a thread-based implementation is limited by the portability o the threads packae used. In the past, when no standardized threads packaes existed, this was a major problem. However, with standard threads packaes such POSIX-compliant pthreads becomin more common, this portability problem with a thread-based implementation is expected to diminish Perormance There are our perormance indices we have measured: throuhput, averae broadcast delivery time, averae broadcast stability time, and averae number o messaes exchaned per atomic broadcast. Throuhput measured rom the event-based implementation is hiher than the throuhput measured rom the threadbased implementation or all atomicity and order semantics (See Fiures 1, 2, and 3). The dierence in the throuhput between the two implementations increases with increase in update arrival rate until a saturation point is reached. We measured averae delivery times or our dierent update interarrival times. The averae delivery time measured rom the event-based implementation is smaller than

4 25 25 WeakTime_event WeakTime_thread StronNO_event StronNO_thread # Updates Arrivin per Second # Updates Arrivin per Second Fiure 1. Throuhput (Wk atm/tm ord). Fiure 3. Throuhput (Strc atm/no ord) StronTotal_event StronTotal_thread # Updates Arrivin per Second Fiure 2. Throuhput (Strn atm/ttl ord). Interarrival Time Event-based Wk atm/ttl ord Strn atm/tm ord Strc atm/no ord Strc atm/ttl ord Thread-based Wk atm/ttl ord Strn atm/tm ord Strc atm/no ord Strc atm/ttl ord Table 1. Delivery time (msec) the averae delivery time measured rom the thread-based implementation or all atomicity and order semantics or all our update interarrival times (See Table 1). Similarly, the averae stability time measured rom the event-based implementation is smaller than the averae stability time measured rom the thread-based implementation or all atomicity and order semantics or all our update interarrival times(see Table 2). Finally, the averae number o messaes exchaned per atomic broadcast is slihtly hiher in the event-based implementation than in the thread-based implementation (See Table 3). The perormance o the event-based implementation is almost always better than the thread-based implementation, except in case o number o messaes exchaned. There are two sources o perormance overhead in the thread-based implementation that are not present in the event-based implementation: context switchin times between threads durin schedulin and the execution time o synchronization primitives. On the other hand, there is one source o peror- mance overhead in the event-based implementation that is not present in the thread-based implementation: the execution time o the select() system call. An event occurrence results in a sinle execution o pthread mutex lock();pthread mutex unlock(); and at least one thread context switch in the threadbased implementation. An event occurrence results in the execution o zero or one select() system call in the event-based implementation. The thread context switchin time is about 16 microseconds (averaed over 1, pthread yield() operations). The execution time o pthread mutex lock(); pthread mutex unlock(); is about 14 microseconds (averaed over the execution o 1, such pairs). The execution time o the select() system call is extremely small (less than 5 micro seconds). These measurements clearly show that the perormance overhead due to the execution o the select() system call is ar less than the perormance overhead due to the execution o pthread mutex lock();

5 Interarrival Time Event-based Wk atm/no ord Wk atm/ttl ord Strn atm/tm ord Strc atm/no ord Thread-based Wk atm/no ord Wk atm/ttl ord Strn atm/tm ord Strc atm/no ord Table 2. Stability time (msec) Interarrival Time Event-based Thread-based Table 3. # o messaes, (Wk atm/no ord) pthread mutex unlock(); and one thread context switch. Perhaps, the dierence in the perormance between the two implementations can be reduced by a much more eicient implementation o a pthreads library. However, we believe that there are some undamental overheads in the thread-based implementation that can t be avoided, and the perormance o an event-based implementation o a roup communication service is always expected to be better. service, such as low control or establishin messae stability. Perhaps, addin/removin eatures in an experimental system is simpler with threads than with events. Reerences [1] Y. Amir, L. Moser, P. Melliar-Smith, D. Aarwal, and P. Ciarella. The totem sinle-rin orderin and membership protocol. ACM TOCS, 13(4): , [2] K. Birman, A. Schiper, and P. Stephenson. Lihtweiht causal and atomic roup multicast. ACM TOCS, 9(3): , Au [3] F. Cristian, S. Mishra, and G. Alvarez. Hih-perormance asynchronous atomic broadcast. Distributed Systems Enineerin, 4(2):19 128, Jun [4] C. Fetzer and F. Cristian. Fail-awareness in timed asynchronous systems. In 15th ACM PODC, Philadelphia, PA, May [5] S. Mishra, C. Fetzer, and F. Cristian. The timewheel asynchronous atomic broadcast protocol. In 1997 PDPTA, paes , Las Veas, NV, Jun [6] S. Mishra, C. Fetzer, and F. Cristian. The timewheel roup membership protocol. In Proceedins o the Workshop on Fault Tolerant Parallel and Distributed Systems, Orlando, FL, Apr [7] S. Mishra, L. Peterson, and R. Schlichtin. Consul: A communication substrate or ault-tolerant distributed prorams. Distributed Systems Enineerin, 1(2):87 13, Dec [8] R. van Renesse, K. Birman, R. Friedman, M. Hayden, and D. Karr. A ramework or protocol composition in horus. In 14th ACM Symposium on Principles o Distributed Computin, Au Conclusion Both the thread-based and the event-based implementation techniques have been used extensively by researchers to implement roup communication services. Our evaluation o these techniques suests that an event-based implementation is preerable to a thread-based implementation. An event-based implementation results in better perormance, simpler, more manaeable, and portable code. It is worth notin that threads have been used in implementin a number o popular roup communication services. There are two reasons that we can think o or the use o threads in implementin these services. First, a threads-based implementation is oten considered a natural way o implementin systems that have concurrent events. This was the main reason why we used threads to implement roup communication services in the past [7, 3]. Second, most o the roup communication systems that use threads are desined to experiment with dierent eatures o a roup communication