EARLY DRAFT. Efficient caching in distributed persistent stores. 1. Introduction. Bengt Johansson *

Transcription

1 Efficient caching in distributed persistent stores. EARLY DRAFT Bengt Johansson * Abstract: This article shows how techniques from hardware systems like multiprocessor computers can be used to improve the efficiency of distributed software, such as distributed persistent stores. It also describes the practical implementation of a particular caching protocol for a distributed store. 1. Introduction A persistent store is an abstraction of a persistent memory, i.e. a memory in which data remains when the process using it has terminated. Therefore a distributed persistent store can be seen as a distributed persistent memory. In other words, a memory that is distributed over several computer and remains in storage when no processes are active. Often this memory is thought of as a shared memory, since the user is presented with a view of one, often monolithic, block of memory. The system we describe uses a client/server-model to implement the store. One or more clients communicate with servers on which the persistent data are stored. (See fig. 1) The servers may reside on the same machine as the clients or on another machine on the network, possibly far away from the clients. s maintain a set of uniquely identified objects, all of which may contain references to other objects in the same, as well as remote stores. Clients perform operations on the stores using remote procedure calls, implemented on top of TCP/IP. Therefore clients and servers may also reside on machines on the Internet. Since the clients use a possibly very slow network to communicate with the servers it is important to decrease the size of the transmitted data. Also, due to the high latency in the network and the relatively small message sizes, the number of individual messages sent must be kept low. Two solutions to this problem is caching of objects and pre-fetching of adjacent objects to the ones requested. * Department of Computing Science, Chalmers University of Technology and Göteborg University bengtj@cs.chalmers.se, WWW:

2 Client Client Client Figure 1. An example of a distributed persistent store. 2. consistency Introducing caches to a system inevitably increases its complexity, since the protocol used must insure that the clients have a consistent view of the system. In other words, they must see updates performed by other clients in some well-defined order. This property is called consistency. Ideally, a system using caches should maintain the same semantics as a system without caches. Accesses to the system are serialised and updates are seen immediately by the other clients. A system satisfying this property is said to be sequentially consistent. [1] However, for a system with caches to be sequentially consistent, the clients must immediately be informed of any updates to the stores. Also, after an update, all other clients must have been informed of it. When the store is updated, the server must immediately send notifications to any other clients that cache the updated object. (See Fig. 2) Therefore each update gives rise to 2*#clients messages, which in most cases is unacceptable. Client1 Write(5) Upd(5) Client2 Figure 2. The messages necessary to perform an update. In distributed multiprocessor systems this problem is, to some degree, solved by relaxing the requirements to keep the cache consistent. One such relaxed consistency model is processor consistency [2]. In this model, client x sees the updates performed by client y in the same order as client y performs them. However, the updates of client x and y together may not be seen by other clients, or x and y themselves, in the same order as they are performed. Using processor consistency it is not necessary to acknowledge the update messages sent to clients. (See Fig. 3) This only reduces the messages to 2+#clients-1, but the server may return the acknowledgement to the updating client without waiting for the other clients to update their

3 caches. However, clients now may have different views of the system for a short but undefined amount of time. Client1 Write(5) Upd(5) Client2 Figure 3. The messages in a system satisfying processor consistency If a system allows for some kind of synchronisation, for instance monitors, transactions or object locking, it is not necessary to maintain the cache consistency for objects that only one user is able modify. The cache consistency is then restored when the user leaves the critical region or unlocks the objects. The weak consistency model [3] is based on the idea that it is possible to identify the points where the system needs to be consistent. For ordinary modifications, (read and write), the caches are allowed to become inconsistent and the system is brought back to a consistent state only at so called synchronisation points. (See fig. 4) Client 1 R W R W Synch+ Updated objects Client 2 Figure 4. Messages in a system satisfying weak consistency 3. Implementation The distributed persistent store described in this article implements a cache protocol satisfying weak consistency. The distributed store is implemented as a client/server application on top of an existing local persistent store. The clients basically provide the same functionality as a local store, extended with functions to manage global references, etc. A client consists of an interface to the user of the store and a transport layer that converts the calls made by the user into messages sent over the network. The transport layer also receives responses from the server and returns them to the user. The server waits for messages from the client and performs the corresponding operations. It then returns a response to the client. One such loop exists for each client that is connected to the server.

4 The client also maintains a cache. When an object is fetched from the server it is stored in the cache. Is stays there until the cache fills up or the server tells the client to remove or update it. An object is removed from the cache when it is locked by another client. The server maintains weak cache consistency by buffering update messages until a synchronisation point, lock or unlock operation, is reached or the buffer is full. When a client reads an un-cached object, the buffer is filled, breadth-first, with objects in the transitive closure of references starting at the requested object. The buffer is then sent to the client. Fig. 5 shows an overview of the system. Client Interface Transport layer Buffer Transport layer Local store Figure 5. Overview of the system When the user makes a call to the store, it goes through the following steps: 1. The client first checks if there are any messages from the server waiting to be processed. Update or remove messages for the cache may have arrived between two calls made by the user. If so, the requested operations are performed on the cache. 2. If the user made a read request, the client checks the cache. If the object is found in the cache, it is returned to the user. 3. If the object is not in the cache, or if the operation is not a read, a message is composed and sent to the server. The client then waits for the response. 4. The server receives the message, decodes it and performs the operation on the local store. 5. If the store is updated, an update message is put in the buffers of the other clients. 6. If a synchronising point is reached, the buffers are sent to the clients. The server then composes a response and sends it back to the client. 7. The client receives the response and returns it to the user. Operation(args) { if(incommingmessgs()) ProcessMessgs(); if(isin(args)) return (args) else { msg=createmsg(op,args); SendMsgTo(msg); rsp=waitforresp(); ProcessResponse(rsp); return Data(resp); (a) Loop() { while(connectionisopen()) { msg=waitformessage(); rsp=decodeandperformop(msg); PutRespInBuffer(rsp); if(isupdated()) { LeaveUpdMsgInOtherBuffers(); if(isreadop()) FillMyBufferWithData(); SendBuffer(); (b) Figure 6. Pseudo-code for the clients and servers

5 Fig. 6 shows the pseudo-code for (a) clients and (b) servers. An example interaction between clients and a store is shown in fig. 5: Suppose client 1 first writes 5 into object x and then 9 into object y. Thereafter object x is unlocked. When the unlocking operation is performed, the buffer is sent to client 2. This process is shown in fig. 5. Client 1 Write(x:=5) Write(y:=9) Unlock(x) x=5 x=5 y=9 Buffer Client 2 x=4,y=8 x=5 y=9 x=5,y=9 Figure 7. An example execution 4. Preliminary results The results presented here are preliminary, but should give some indication of the benefits of caching in a system like the distributed persistent store. The test is a simple producer-consumer system, where the producer generates a list of values and the consumer reads the values as they soon as they become available to it. Table 1 shows the speed-up resulting from using caches in the system. The times presented are the averaged execution times on an Sun Ultra-1 140MHz with 64Mb RAM. Without cache With cache Speed-up 8.18s 3.81s 2.14 Table 1. The speed-up achieved using caches. Table 2 shows the execution time depending on the buffer size. The speed-up achieved here is mainly due to the fact that the pre-fetching increases the cache hits. The table shows the execution times with cache and different buffer sizes. All processes are run on the same machine as for Table 1. Note that using a 0K buffer the system satisfies processor consistency, since Buffer size Execution time Processor usage hits no cache 70.3s 44.9% - 0K 75.7s 49.7% 0% 1K 295.9s 0.2% 88.2% 4K 27.1s 62.1% 98.4% 16K 26.7s 65.8% 99.6% 64K 26.1s 63.7% 99.9% 256K 25.4s 64.6% 99.96% Table 2. Clients and server running on the same machine such a buffer immediately fills up and is sent to the clients. Table 3 shows the same algorithm but the server is running on another machine. The bad times, in both cases, for 1K buffers are due to congestion in the network.

6 Buffer size Execution time Processor usage hits 1K 297.1s 0.5% 88.2% 4K 95.5s 2.4% 96.9% 16K 44.1s 36.3% 99.2% 64K 26.6s 65.2% 99.8% 256K 25.8s 65.5% 99.95% Table 3. running on another machine These results do not show the benefits of using relaxed cache consistency in the system. Therefore, further testing is necessary to get definitive results. 5. Related work Multiprocessor computers, especially those with distributed memory, often take advantage of relaxed cache consistency models. Not only do they benefit from the decreased data-flow over their interconnections, relaxed models also allow compiler writers to perform code-optimizations that would respect the semantics of the source-program on a single processor machine, but would break down in a multiprocessor with a sequentially consistent cache. The processor consistency model [2], allows writes from two or more processors to be observed in different order on different processors. This model is implemented in the VAX multiprocessor. Weak consistency [3], takes advantage of the fact that many memory updates are performed in critical sections, where only one processor may access the data. Therefore it is unnecessary to enforce a strict consistency model except when entering or leaving critical sections. The weak consistency model distinguishes between ordinary accesses and synchronising accesses, at which cache consistency is ensured. For an overview of cache consistency models see [4]. The work in distributed persistent stores have so far not concentrated on improving efficiency in the model, but rather to show that persistent stores and the persistent programming model has advantages compared to relational databases or remote object invocation in CORBA. Examples of distributed persistent stores and operating systems are PerDis [5] and Grasshopper [6]. 6. Further work The implementation presented in this article can be further improved. At the moment all writes are immediately performed, even if they are to a locked object. This situation can be improved by not having a write-through cache in the clients. Updates can be performed globally at synchronisation points. Update messages are sent to all clients connected to a particular server even if the client is not using the object. Therefore it is possible to avoid sending unnecessary messages to clients if the server keeps a list of the cached objects on each client. The server then only propagates messages to clients that have a copy of the object. We intend to explore alternative implementations of the protocol. For instance, it is possible to have replicated caches in the server instead of buffers.

7 7. Conclusions Relaxed consistency models are much used in multiprocessor implementations. They lead to decreased memory latency and makes it possible to take advantage of program optimization in compilers and in hardware. So far distributed persistent stores have not made use of such models, but we have shown that weak consistency gives an increase in performance when employed in a software system such as a persistent store. 8. References [1] Leslie Lamport, How to make a multiprocessor computer that correctly executes multiprocess programs, IEEE Transactions on Computers, C-28(9): September [2] James R. Goodman, consistency and sequential consistency, Technical Report no. 61, SCI Committee, March [3] Michel Dubois, Christoph Scheurich and Fayé Briggs, Memory access buffering in multiprocessors, In Proceedings of the 13th Annual International Symposium on Computer Architecture, pp , June [4] Kourosh Gharachorloo, Daniel Lenoski, James Laudon, Phillip B. Gibbons, Anoop Gupta and John L. Hennessy, Memory Consistency and Event Ordering in Scalable Shared-Memory Multiprocessors, ISCA 1990: [5] Paulo Ferreira, Marc Shapiro, Xavier Blondel, Olivier Fambon, João Garcia, Sytse Kloosterman, Nicolas Richer, Marcus, Roberts, Fadi Sandakly, George Coulouris, Jean Dollimore, Paulo Guedes, Daniel Hagimont, and Sacha Krakowiak, PerDiS: design, implementation, and use of a PERsistent DIstributed, Tech.Report: QMW TR752, CSTB ILC/ , INRIA RR 3525, INESC RT/5/98, URL: www-sor.inria.fr/publi/pdiupds_rr3525.html, October 1998 [6] Alan Dearle, Francis Vaughan, Rex di Bona, James Farrow, Frans Henskens, Anders Lindström and John Rosenberg, Grasshopper: An orthogonally persistent operating system, Tech.Report GH10, Dept. of Computer Science University of Adelaide, URL: 1994