Symmetric Tree Replication Protocol for Efficient Distributed Storage System*

Transcription

1 ymmetric Tree Replication Protocol for Efficient Distributed torage ystem* ung Cune Coi 1, Hee Yong Youn 1, and Joong up Coi 2 1 cool of Information and Communications Engineering ungkyunkwan University , uwon, Korea {coisc,youn}@ece.skku.ac.kr 2 Information ecurity Tecnology Division Korea Information ecurity Agency , eoul, Korea jscoi@kisa.or.kr Abstract. In large distributed systems, replication of data and service is needed to decrease te communication cost, increase te overall availability, avoid single server bottleneck, and increase te reliability. Tree quorum protocol is one of te replication protocols allowing very low cost in te best case but as some drawbacks suc tat te number of replicas grows rapidly as te level increases and root replica becomes a bottleneck. In tis paper we propose a new replication protocol called symmetric tree protocol wic efficiently solves te problems. Te proposed symmetric tree protocol also requires muc smaller cost tan te previous protocols. We conduct cost and availability analysis of te protocols, and te proposed protocol displays comparable availability to te tree protocol using muc smaller number of nodes. It is tus effective to be applied to survival storage system. Keywords: replication protocol, tree quorum, availability, distributed system, symmetric tree 1 Introduction In large distributed systems node and communication failures are likely to occur wic can prevent te operations from being successfully carried out. Tis raises te need for introducing a sufficient level of fault tolerance into te distributed systems [1]. Oter problems are increased communication cost as te system grows and bottleneck of a single server as many processes compete for te same service. Replication of data and services is one of te practical solutions to te problems. It can avoid te server bottleneck problem and increase overall availability [2]. However, te communication cost increases as te number of replicas increases. In order to minimize te *Tis work was supported by Korea Researc Foundation Grant (KRF D00421). Corresponding autor: Hee Yong Youn P.M.A. loot et al. (Eds.): ICC 2003, LNC 2659, pp , pringer-verlag Berlin Heidelberg 2003

2 ymmetric Tree Replication Protocol for Efficient Distributed torage ystem 475 communication cost, tus, te number of replicas sould be kept as small as possible [3,4]. We consider a distributed system in wic data are fully replicated and two types of operations are allowed on te replicated data, namely and write. A operation returns some value wile write operation installs a new value. Proper syncronization is acieved if operations return te value installed by te last write operation, and and write operations or two write operations are not executed concurrently. Eac node uses a centralized concurrency control sceme to syncronize te accesses to its local copy and a replication control protocol to coordinate te accesses to various replicas. Te basic principle employed in maintaining consistency of replicated data is to require conflicting operations to lock at least one common copy. A distributed system consists of a set of distinct sites tat communicate wit eac oter by sending messages over a communication network. No assumptions are made regarding te speed, connectivity, or reliability of te network. We assume tat te sites are fail-stop. A distributed database consists of a set of objects stored at several sites. Users interact wit te database by invoking transaction programs. Transactions are partially ordered sequence of and write operations tat are executed atomically. Execution of a transaction must appear atomic, i.e. a transaction eiter commits or aborts. In a replicated database, copies of an object may be stored at several sites in te network. Multiple copies of an object must appear as a single logical object to te transactions. Tis is termed as one-copy equivalence and enforced by replication control protocol. In te literature tere exist several replication algoritms aiming at different goals. Te Read-One/Write-All algoritm [5] as minimum cost and maximum availability but as maximum communication cost for write operation. Te quorum consensus [6] and dynamic voting [7] ave good and write availability but as a disadvantage of ig cost. For all tese replication control protocols, making te operation ceaper ends up wit a more expensive write operation and vice versa. All te strategies ave a cost of O(n), wic means te operation cost linearly depends on te number of replicas in te system [8]. By assigning a logical structure to a set of replicas, it is possible to reduce te communication cost. Multi-Level-Voting-Protocol [9], Weigted Voting [10], Tree quorum protocol [11] and Grid protocol [12] are suc type of replication protocols, but tey still ave relatively ig operation cost especially wen te number of replicas is large and some failures exist. Te tree quorum protocol as cost of 1 in te absence of failures and provides graceful degradation wen failures exist. Hence in tis paper we develop a new replication protocol wic as low operation cost. We compare te replication protocols wic use a logical tree structure and te proposed symmetric tree structure. Te proposed symmetric tree protocol allows comparable availability to te tree protocol wit muc smaller number of nodes. It is tus effective to be applied to survival storage system. Te rest of te paper is organized as follows. In ect. 2 we describe te earlier protocols, and in ect. 3 we propose a new protocol. We compare te new symmetric tree protocol wit earlier protocols in ect. 4. ection 5 concludes te paper.

3 476. C. Coi, H.Y. Youn, and J.. Coi 2 Review of Earlier Protocols 2.1 Logaritmic Replication Protocol Te class of replication protocols discussed ere is a generalization of te Tree Quorum Protocol. Tese replication protocols use a logical tree structure. All te algoritms ave cost of 1 in te absence of failures and provide graceful degradation wen failures exist. Te replication protocols wic use tree structure ave varying costs and availabilities according to fault condition, wereas oter replication protocols ave constant costs and availabilities. Let n be te number of replicas wic can be organized as a tree of eigt. A non-leaf replica R i as Ri descendants. For eac replica except te leaves we define quorum rq Ri, and a write quorum wq Ri. A tree consisting of 13 replicas organized in tree levels is sown in Fig. 1. Algoritm: Te algoritm is recursive: A tree of eigt 0 consists only of a leaf replica and can be locked by simply locking te replica for eiter or write operation. Read operation: Locking a tree of eigt for a operation means to lock te root replica or rq ROOT of its subtrees wit te rq ROOT descendants of te root serving as new root replicas of te subtrees of eigt -1. Write operation: Locking a tree of eigt for a write operation means to lock te root replica and wq ROOT of its subtrees. Here, te wq ROOT descendants of te root serve as new root replicas for te subtrees of eigt -1. R1 WQ=1,RQ=3 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 R13 Fig. 1. A tree of eigt 3 wit 13 replicas Examples of valid quorum sets (RQ) from Figure 1 are {R1}, {R2, R3, R4} wen node-1 is not available, and {R5, R6, R7, R3, R11, R12, R13} wen node-1, node-2, and node-4 are not available. Examples of valid write quorum sets (WQ) are {R1, R2, R5}, {R1, R3, R10}, and {R1, R4, R12}. In order to maintain consistency of te replicated objects, te selected and write quorums ave to fulfill some requirements. To detect /write conflict, any

4 ymmetric Tree Replication Protocol for Efficient Distributed torage ystem 477 valid quorum set must intersect wit valid write quorum set. Tis requirement can be satisfied by rqri + wqri > VRi were V Ri is te number of descendants. To detect write/write conflicts, any two valid write quorum sets must intersect. Tis is always satisfied because every valid write quorum set includes te root. o tere exist many cases of valid RQ and WQ in te tree quorum protocol. After reviewing all possible cases regarding te provided cost and availability, te strategy wit RQ = and WQ = 1 is selected as te best coice. Te case is called Logaritmic Protocol. 2.2 Grid Protocol Here te replicas are logically arranged in a grid topology, and and write operations are required to lock te rows and columns of nodes suc tat conflicting operations request a common node. In tis protocol quorum consists of a node from every column and write quorum consists of a node from every column plus a full column of nodes. A grid network wic consists of 4 rows and 4 columns of 16 replicas is sown in Fig. 2. Algoritm: Read operation: elect and lock any single node from eac column. Tis set of nodes is called Column-cover (C-cover). Write operation: Coose a C-cover and lock any wole column. R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 R13 R14 R15 R16 Fig. 2. A Grid network wit 16 replicas Examples of valid quorum sets ere are {R1, R6, R3, R12} and {R1, R6, R7, R8}, and examples of valid write quorum sets are {R1, R10, R3, R4, R2, R6, R14} and {R9, R6, R3, R8, R7, R11, R15}. Here two write operations cannot be executed concurrently because eac one locks a C-cover and all nodes in a column. If two operations ave locked a C-cover, neiter one can obtain write-locks from all te nodes in any column. imilar argu-

5 478. C. Coi, H.Y. Youn, and J.. Coi ment can be used to sow tat concurrent execution of and write operations are not possible. Te grid protocol as constant operation cost because te size of quorum set is always same. One disadvantage of tis protocol is te operation cost is equally ig regardless of failure condition. However, it as iger availability tan tree quorum protocol. To solve te problems of te tree quorum and grid protocol, we next present te proposed symmetric tree protocol. 3 Te Proposed Protocol As mentioned in te previous section, tree quorum protocol and grid protocol ave some drawbacks, respectively. In tis section we propose a new replication protocol called symmetric tree protocol to solve te drawbacks of te previous protocols. 3.1 Motivation In typical replication protocols almost all replicas need to be accessed for and write operation. For example, in Quorum consensus protocol of 16 replicas, quorum + write quorum need to be greater tan 16. However, if we assign a logical structure to te replicas, ten we can reduce te number of replicas because we can organize te structure in suc a way tat te tree and grid topology are used togeter. As previously described, a major weakness of tree quorum protocol is tat te number of replicas rapidly grows as te tree level grows. Te tree quorum protocol as cost of 1 in te absence of failures. As a result, root replica can become a bottleneck. To solve tis problem, te proposed protocol as two root replicas using symmetric tree topology. Terefore, te proposed protocol can ave te advantage of aving minimum operation cost weter failures exist or not. Also it as te advantage of aving not many replicas in ig levels like grid protocol. 3.2 Te Proposed ymmetric Tree Protocol Te proposed protocol solves te problem of oter replication protocols of significantly increased cost wen failures exist using symmetric tree topology. It as te advantage of aving minimum operation cost in te best case like tree structure protocol and also as te advantage of aving not so many replicas in iger levels like grid protocol. In tis protocol replicas are organized as sown in Figure 3, were two tree structures are joined wose leaf nodes are common. We call te two trees uptree and down-tree, respectively. Algoritm: Read Operation: Read operation starts from eiter te root node of up-tree or tat of down-tree. If te root replica is not available, all te descendants of it are. Eac te descendant

6 ymmetric Tree Replication Protocol for Efficient Distributed torage ystem 479 serves as a new root replica for te subtree to te /2-level of te symmetric tree, were is te number of levels of te tree. After te /2-level, te next level is if any node of te parent nodes is not available. Write Operation: Write operation starts from any root replica of te two trees. Only one replica of te descendants of te root must writes. A descendant of te root serves as a new root replica of te subtree. Te descendant of te parents must write from te /2-level to te last level. R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 R13 R14 R15 R16 R17 Fig. 3. A network for te proposed symmetric tree protocol wit 17 replicas From Fig. 3 examples of valid quorum set are {R1}, {R2, R3, R4}, {R3, R4, R5, R6, R7}, {R3, R4, R14}, {R1, R8, R9, R10, R16}, {R14, R15, R16}, and {R17}. Examples of valid write quorum sets are {R1, R2, R5, R14, R17}, {R1, R3, R9, R15, R17}, and {R1, R3, R10, R15, R17}. Consistency is maintained in selecting, write quorum set to satisfy te requirement of detecting /write conflict and write/write conflict. Te Read/Write conflict can be detected because operation sould lock te wole descendants and write operation sould lock at least one replica of wole descendants. Te Write/Write conflict can also be detected because every write operation as to lock te root replica of te symmetric tree structure. Tus, if one replica is locked, ten oters cannot get te lock. 4 Performance Analysis In tis section cost and availability of te proposed protocol and tree protocol are analyzed.

7 480. C. Coi, H.Y. Youn, and J.. Coi 4.1 Cost Analysis For cost analysis, cost and write cost are considered, wic are computed by te number of sites involved in te operation. A operation in bot te protocols need to consult only te root replica wic is very efficient if root replica is available. However, te root replica becomes a bottleneck if all operations are done on it. Terefore te algoritm is sligtly modified as te level of te tree te operation will be performed is randomly selected. For tis, uniformly distributed random variable X in te interval [0,1] and a parameter f in interval [0,1] are cosen. A random value x of X is generated and te top-level is cosen for performing operation if x f. Oterwise, a new x is generated and te next level is cosen if x f, etc Logaritmic Protocol uppose tere are +1 levels. Te average cost computes to C = f + (1 f ) f RQ + (1 f ) f RQ + L + (1 f ) f RQ = f (1 f ) RQ + 1 f (1 f ) RQ k= 0 k= k k k (1 ( f + (1 f) f + (1 f) f + L + (1 f) f)) RQ (1) In te tree protocol, we consider Logaritmic protocol wic is te best coice regarding te cost and availability. Te Logaritmic Protocol is defined by a quorum of RQ = and write quorum of WQ = 1. Average cost is min( C ) = 1 (2) ((1 f) ) 1 C = f + ((1 f ) ) (1 f) 1 (3) For write operation, Cwrite = + 1 (4) ymmetric Tree Protocol For cost analysis, cost and write cost are considered, wic are computed by te number of sites involved in tat operation. A and write operation in bot te protocols only need to consult te root replica wic is very efficient if root replica is available. Average cost is computed by

8 ymmetric Tree Replication Protocol for Efficient Distributed torage ystem C = f + (1 f) f + (1 f) f + L+ (1 f) f + (1 f) f + L+ (1 f) f + (1 ( f + (1 f) f + L+ (1 f) f)) k k k k 2 2 f (1 f) (1 f) k = f (1 f) k= 0 k= 1 k = 0 (5) min( C ) = 1 (6) For write operation, Cwrite = + 1 (7) Comparison of average cost for bot te protocols are sown in Fig. 4 for 121 nodes of tree quorum protocol and 161 nodes of symmetric tree protocol. Average write cost for tem is depend on eigt of tree. Read Cost Logaritmic Protocol (121 nodes) ymmetric tree Protocol (161 nodes) p Fig. 4. Comparison of average cost Te tree quorum protocol consists of 121 nodes of 5 levels and te last level as te maximal number of node. On te oter and, te symmetric tree protocol consists of 161 nodes of 9 levels and te fourt level as te maximal number of node. Also, te maximal number of node is te same. Observe from Figure 4 tat cost of te proposed protocol is always muc smaller tan logaritmic protocol in sprite of as more nodes. Te difference between te two protocols gets more substantial wen te failure of nodes is low.

9 482. C. Coi, H.Y. Youn, and J.. Coi 4.2 Availability Analysis Logaritmic Protocol Te availability of operation for te tree quorum protocol performed on a tree of eigt l > 0 is k () l ( l1) ( l1) k p (1 p) ( k )( ) ( 1 ) k= RQ = + wit = p (8) (0) o te availability of operation for te logaritmic protocol is l ( ) () l ( 1) = p+ (1 p) since RQ = (9) Te availability of write operation is ( )( ) ( 1 ) () l ( l 1) k ( l 1) k write p k write write k= WQ (0) = wit write = p (10) o te availability of write operation for te logaritmic protocol is l ( 1 ( 1 write ) ) = p (11) () l ( 1) write ymmetric Tree Protocol Te availability of operation for te proposed protocol performed on eigt / 2 l > 0 is ( ) ( ) ( ) () l ( l (1 ) 1) 1 ( l 1) = p+ p + p wit (0) = p (12) Te availability of write operation is ( )( ) ( 1 ) () l 2 ( l 1) k ( l 1) k write p k write write k= WQ = (0) wit write = p (13) Comparisons of availabilities of te two protocols are sown in Fig. 5. For te comparison, we examine 3 cases, eac one wit a different number of replicas. Tey are 17 of level 5, 53 of level 7, and 161 replicas of level 9. As illustrated in Fig. 5, te availability of te proposed protocol increases as te level of tree increases. Note tat te availability of symmetric tree protocol is iger tan logaritmic protocol wit muc smaller number of nodes.

10 ymmetric Tree Replication Protocol for Efficient Distributed torage ystem 483 Logaritmic Protocol (121 nodes) ymmetric Tree Protocol (53 nodes) ymmetric Tree Protocol (17 nodes) ymmetric Tree Protocol (161 nodes) Read Availability p Fig. 5. Comparison of availability 5 Conclusion In tis paper we ave proposed a new symmetric tree replication protocol wic is better tan Logaritmic protocol in cost and availability using muc smaller number of nodes. One of te main advantages of te proposed symmetric tree protocol is muc smaller cost tan previous protocols even toug te failure rate of te nodes increases. Te coice of wic replication protocol sould be used depends on te primary performance criterion, and te new proposed symmetric tree protocol can be selected for low operation cost and ig availability. As future work, we will analyze and compare te response time and trougput of te proposed protocol. Implementation of te protocols for survivable storage system will also be carried out. Reference [1] C. Amza,, A.L. Cox, W. Zwaenepoel, Data replication strategies for fault tolerance and availability on commodity clusters, Proc. Int l Conf on Dependable ystems and Networks (DN), 2000, [2] H.Y. Youn, B. Krisnamsetty, D. Lee, B. K. Lee, J.. Coi, H.G. Kim, C.W. Park, and H.. Lee, An Efficient Hybrid Replication Protocol for Higly Available Distributed ystem, Proc. Int l Conf on Communication and Computer Networks (CCN), Nov, 2002, [3] K. Arai, K. Tanaka, M. Takizawa, Group protocol for quorum-based replication, Proc. event Int l Conf on Parallel and Distributed ystems, 2000, [4] G. Alonso, Partial Database Replication and Group Communication Primitives, Proc. of te 2 nd European Researc eminar on Advances in Distributed ystems (ERAD 97), Marc 1997, [5] P.A. Bernstein and N. Goodman, An Algoritm for Concurrency Control and Recovery in Replicated Distributed Databases, ACM Trans on Distributed ystems, 9(4), 1984,

11 484. C. Coi, H.Y. Youn, and J.. Coi [6] R.H. Tomas. A Majority Consensus Approac to Concurrency Control for Multiple Copy Data-based, ACM Trans on Database ystems, 4(2),1979, [7] D. Davcev, A Dynamic Voting ceme in Distributed ystems. IEEE Trans on oftware Engineering, 15(1), 1989, [8] D. aa,. Rangarajan,.K. Tripati, An Analysis of te Average Message Overead in Replica Control Protocols, IEEE Trans on Parallel and Distributed ystems, 7(10), Oct. 1996, [9] B. Freisleben, H.H. Koc, and O. Teel, Designing Multi-Level Quorum cemes for Higly Replicated Data. Proc. of te 1991 Pacific Rim Int l ymp on Fault Tolerant ystems, IEEE, 1991, [10] D.K. Gifford, Weigted Voting for Replicated Data, Proc. of te 7 t ACM ymp on Operating ystems Principles, 1979, [11] D. Agrawal and A. El Abbadi, Te tree Quorum protocol: An Efficient Approac for Managing Replicated Data, Proc of te 16t Very Large Databases (VLDB) Conf, 1990, [12]. Ceung, M. Ammar, and M. Aamad, Te Grid Protocol: A Hig Performance ceme for Maintaining Replicated Data, Proc of te 6t Int l Conf on Data Engineering, 1990,