Increasing the Accuracy of Cost and Availability Predictions of Quorum Protocols

Transcription

1 2017 IEEE 22nd Pacific Rim International Symposium on Dependable Computing Increasing the Accuracy of Cost and Availability Predictions of Quorum Protocols Robert Schadek and Oliver Theel Carl von Ossietzky University of Oldenburg, Germany Department of Computer Science Abstract Highly available services can be implemented by means of so-called quorum protocols. Unfortunately, applying quorum protocols in real-world physical networks turns out to be difficult since efficient quorum protocols often depend on a particular graph structure imposed on the replicas managed by it. In this paper, we show that the cost and availability predictions of the operations provided by quorum protocols are often not accurate when they are executed in a real-world physical network. We present the mapping approach, the mapping approach increases the accuracy of the cost and availability predictions of most quorum protocols on real-world networks. Additionally, we use this mapping approach to analyze the resulting operation availability of two commonly known quorum protocols when applied to a physical network. Keywords Distributed Systems, Fault Tolerance, Data Replication, Quorum Protocols, Operation Availability Prediction, Physical Network Topologies I. INTRODUCTION Highly available access to a data object is a core problem in the field of computer science. Relying on a single replica of a data object greatly limits the availability of this data. Therefore, the consequence is to increase the number of replicas storing the data object. This increases the availability of the data object as it can be accessed using different replicas. Replicating the data object introduces the need for synchronization. Let the data object be replicated on five replicas, as shown in Figure 1. If the data object located on replica 0 is updated with a new Figure 1: Five replicas of a data object. value, then reading the data object from replica 4 does not yield the new value. Obviously, that is not the intended behavior. An additional problem is that two concurrent write operations can be executed on different replicas on the same data object at the same time. For example, value a is written to the data object on replica 2 and value b is written to the data object on replica 3. This raises the following question: Which value is the correct one? Both examples show that simply creating multiple replicas of a data object does not necessarily lead to the expected results. Usually, the goal is that all operations behave as they would do on a non-replicated data object. More formally, this non-replicated behavior can be achieved by a control protocol that guarantees the one-copy serializability (1SR) property [1]. In general, quorum protocols (QPs) provide highly available access to data by means of replication and at the same time maintain this 1SR property. In most cases, QPs provide a read and a write operation to read and write data. These QPs use read quorums (RQs) and write quorums (WQs) to execute the desired operation. Quorums are specific subsets of replicas of the set of all replicas. Commonly, a read operations reads the data of all replicas of a RQ and identifies an up-to-date replica. Write operations, on the other hand, use an atomic commit protocol, like the Two-Phase commit Protocol [1], to write the new data to all replicas of a WQ. QPs can be classified into unstructured quorum protocols (UQPs) and structured quorum protocols (SQPs). SQPs arrange the individual replicas in a graph structure (GS), and use this GS to formulate rules how to construct quorums. UQPs specify no such GS. In order for read and write operations to work, the data of the operations needs to be communicated to and from the replicas of the quorum used. Implicitly, most UQP and SQP rely on a completely connected GS as a communication medium between its replicas. We call this completely connected GS a logical network topology (LNT). A physical network topology (PNT) is a GS that actually arranges and connects the replicas in the real-world. QPs have no influence on the PNT. In this paper, we show that the PNT used as a communication medium between the replicas, has to be considered in the cost and availability analyses of a QP to improve the prediction accuracy of those analysis. This paper is structured as follows. Section II discusses the system model used as a basis for the analysis. For completeness and ease of understanding, the Majority Consensus Protocol (MCS) [2] and the Triangular Lattice Protocol (TLP) [3] are presented in Section III-A and Section III-B respectively. In Section IV, we present the Mapping Approach. This approach allows to analyze the read and write availability and the read and write costs of any QP on any PNT. The Mapping Approach is then used in Section V to exemplary evaluate the TLP and the MCS on a PNT. It is shown that the cost and availability predictions differ when considering the PNT in comparison to the original analysis. Section VI concludes the paper. It discusses the results of this paper and gives ideas to further increase the prediction accuracy of the cost and availability measurements of QPs when applied to PNTs. II. SYSTEM MODEL In order to analyze QPs, their characteristics, and their use on PNTs, we first need to define a coherent model /17 $ IEEE DOI /PRDC

2 A. Graph Structure A graph structure GS = (V,E) is a two-tuple of a set of vertices V and a set of edges E. Edges connect vertices. V (GS) gives the set of vertices of a GS. E(GS) gives the set of edges of a GS. A vertex v V is a tuple v =(i, c x,c y ), where i N is the ID of the vertex and c x and c y are the coordinates (i.e. its position/location) of the vertex in the corresponding dimensions. The shorthand notation v i gives a vertex with ID i. N = V (GS) represents the number of vertices in a GS. An edge e i,j E is defined as e i,j := (v i,v j ), where v i,v j V. A path v 0,v 1,...,v n between v 0 and v n exists in GS, iff: i, 0 i n : v i V (GS) and (1) i, 0 i<n: e =(v i,v i+1 ) E(GS) (2) If a path exists, then the two nodes v 0 and v n are called connected. V( v 0,v 1,...,v n ) denotes the set of vertices of a path such that: V( v 0,v 1,...,v n ):={v 0,v 1,...,v n }. (3) E( v 0,v 1,...,v n ) denotes the set of the edges of a path such that: E( v 0,v 1,...,v n ):={e 0,1...,e n 1,n }. (4) Each vertex is assumed to be hosting exactly one node which, in turn, is assumed to be hosting one replica of the replicated data object. The terms node and replica will be used synonymously throughout this paper. B. Consistency Criteria In this paper, we discuss QPs that guarantee the 1SR property. Informally, 1SR states that read and write operations on replicated data object have the same observable effect as they would have on non-replicated data [1]. QPs use (b) A GS used by the Triangular (a) A random GS. Lattice Protocol. Figure 2 quorums to execute their operations. Usually, QPs provide a read operation, executed on a RQ, and a write operation, executed on a WQ. Let Q be a QP providing a read and a write operation that upholds the 1SR property. When 1) every RQ of Q intersects 1 with every WQ of Q, 2) all WQ of Q intersect with each other, and 3) replicas of Q can be locked exclusively for reading operations or a writing operation. Q upholds the 1 Two quorums a and b intersect, if a b. 1SR property, because only a single write operation can write all replicas of its WQ or one or more read operations can read their RQs. This quorum intersection approach is used by many QPs to provide the 1SR property. All the QPs discussed in this paper do so as well. C. Fault Model Nodes hosting a replica are assumed exhibit a fail-silent behavior. All failures are independent of each other. The availability of a node is described by p, where 0 p 1. Ap value of 1 means that the node is available with a probability of 100%. Ap value of 0 means that the node is available with a probability of 0%. All nodes are assumed to have the same p value. This simplification gives way to a feasible analysis. Communication channels aka. edges are always available between two available nodes. In other words, if one of the nodes connected by an edge is not available then the edge is also not available. D. Read and Write Costs and Availability The probability that a read or write operation is available for a given QP under the node availability p is described by a r (p) and a w (p) respectively, where 0 a r (p),a w (p) 1. The minimal average costs per read and write operation are given by c r (p) and c w (p). For some QPs, there exist a closed formula to compute a r (p), a w (p), c r (p), and c w (p). In general, the equations presented below, are used to analyze the read and write availability as well as the costs per operation of a QP. Given all RQs and all WQs, and RQS = {c c P(N) x RQs for which c x} (5) a r (p) = p c (1 p) N c. (6) c RQS WQS = {c c P(N) x WQs for which c x} (7) a r (p) = p c (1 p) N c. (8) c WQS are used to calculate the a r (p) and a w (p) for a given p. P(s) denotes the power set of s gives all unique subsets of s [4]. The set of all RQs is used to construct the read quorum set (RQS), which is the set of all replica sets that have a subset which is a RQ for the given QP. Different elements of the RQS can have the same subset in the set of RQ. Analogously, the same is true for the write quorum set (WQS) of a WQ. The Equation 6 and Equation 8 are taking the sum of the probability of all elements in RQS or WQS being available for a given replica availability p. The average read c r (p) and (write costs c w (p)) measure the average minimal number of replicas that is required to execute an operation for a given replica availability of p. It is computed by summing up the minimal number of replicas required to construct a read quorum (or write quorum) for every replica set in RQS (or WQS) with the probability of the replica set appearing. Additionally, this sum needs to be divided by a r (p) 99

3 (respectively a w (p)) to normalize the cost measure. The costs for the operations are given by c RQS c r (p) = p c (1 p) N c minrq(c) (9) a r (p) and c w (p) = c WQS p c (1 p) N c minw Q(c). (10) a w (p) The function minrq(c) (and minw Q(c)) determines the minimal RQ (and WQ) for the given combination of available replicas. III. DISCUSSION OF EXISTING QUORUM PROTOCOLS A. The Majority Consensus Protocol (MCS) The MCS reads N/2 replicas and writes (N +1)/2 replicas, where N is the total number of replicas [2]. MCS guarantees 1SR. The read availability a r (p) of the MCS is N ( ) N a r (p) = p k (1 p) N k (11) k k= N/2 and the write availability a w (p) is N ( ) N a w (p) = p k (1 p) N k. (12) k k= (N+1)/2 The read costs c r (p) are N/2. The write costs c w (p) are (N +1)/2 [5]. As MCS is a UQP, it implicitly assumes that there is a direct communication link between all the nodes. B. The Triangular Lattice Protocol (TLP) Figure 2b shows an example of a GS used by the TLP. Every RQ consists of a complete vertical or a complete horizontal path through the GS. Every WQ consists of a complete vertical and a complete horizontal path through the GS. In Figure 2b, the diagonal path 0, 4, 8 connecting the replicas creates a minimal path that crosses the GS vertically and horizontally. This diagonal path can therefore be used as a very efficient WQ. As quorums for the TLP are created by finding paths through a GS, no simple closed formula exists that calculates the read and write availability. Therefore, we use the formulas presented in Section II-D are used to calculate the read and write availability, and the cost per operation for the TLP [3]. IV. AVAILABILITY COMPARISON In Section II-D, we showed how to compute the read and write availability a r and a w, as well as the average minimal costs c r and c w for any quorum. As discussed in Section I, QPs assume direct communication links between their nodes. In order to usefully and fairly analyze and compare QPs, we have to perform the analysis on a PNT. This is because, in general PNTs do not offer direct communication links between all nodes of the real-world network. Therefore, additional nodes are required to reestablish the communication between the nodes of the original quorum. These additional nodes need to be considered in the analysis as more nodes per operation results in lower availability and higher costs. In Section IV-A, we present the Mapping Approach, where the nodes of the LNT of the QP are mapped to the nodes of the PNT. The GS of the PNT is then used as the communication medium for the analysis. None of the presented QPs has such a mapping feature, nor could such a mapping algorithm be identified in the literature. A. The Mapping Approach A mapping is an injection from one GS to another GS. This requires that the number of nodes in the codomain structure is at least equal to the number of the nodes of the original structure. Formally, a mapping M(GS, GS ) from GS to GS is defined as: M(GS, GS )={(v 1,v 1),...,(v n,v n)} (13) (v, v ) M : v V,v V (14) (v, v ), (v, v ) M : v = v (15) (v,v),(v,v) M : v = v (16) When a mapping has been created, the QP logically works on its LNT. But the mapping is then used as a communication medium between the nodes. For every node selected to participate in a quorum, the mapping selects the mapped node of the PNT. After the QP has selected all necessary nodes to construct a quorum based on its LNT, the mapping tests whether the nodes are connected. As mappings are usually not between homomorphic GSs, in the general case, it can be assumed that the replicas of the quorum are not connected directly with each other in the PNT [6]. Consequently, the mapping adds additional nodes to reestablish the communication between the nodes of the quorum. The cost and availability analysis of a Algorithm 1 Procedure ARENODESCONNECTED 1: Inputs: nodes = a set of nodes that is tested whether they are connected in GS GS = the graph structure the nodes are supposed to be connected in 2: Returns: true if there exists a path between all two elements of nodes in GS, false otherwise 3: procedure ARENODESCONNECTED(nodes, GS) 4: return ( n, n nodes : n,...,n GS) 5: end procedure mapped protocol has to consider the additional nodes required on the PNT level. It is a desired goal, to obtain mappings that require few additional replicas on the PNT level in relation to the LNT level, as thus the expected cost and availability characteristics of the QP on the LNT level can be matched the closest. For that reason, the important question is how to create good or even optimal mappings. In order to rate mappings, we have to define a measurement criterion. Any number of criteria can be selected depending on the intended use of the QP. Possible criteria are the average 100

4 write costs, maximal costs, average costs, write availability, etc. For this paper, we use the average read and write availability value (ARW). The ARW approximates the weighted accumulation of the integral of the read and write availability. ARW = 1.0 p=0.0 a w (p)+ 1.0 p=0.0 The higher the ARW, the better is the mapping. a r (p) (17) B. Optimal Mapping In this section, we present the OptimalMapping algorithm, that finds the optimal mapping under the given measurement criterion. Optimal in the scope of this paper means: the mapping where the ARW is the highest. Algorithm 2 Procedure FINDSMALLEST 1: Inputs: nodest oconnect = the nodes for which a path needs to exists, so they can communicate GS = the graph structure in which the nodest oconnect need to be connected in 2: Returns: the smallest subset of V (GS) for which the nodest oconnect are connected 3: procedure FINDSMALLEST(nodesT oconnect, GS) 4: # all subsets of replicas including nodest oconnect 5: subsets P(V(GS)) 6: smallest V (GS) 7: 8: # the subsets are iterated in increasing order of the 9: # number of replicas contained in each subset 10: for all i subsets do 11: if ARENODESCONNECTED( nodestoconnect, GS(i,E(GS)) ) i smallest then 12: smallest i 13: end if 14: end for 15: return smallest 16: end procedure To get an intuition for the mapping approach, see the following example. Let the TLP of Figure 2b be mapped to the PNT in Figure 2a with the mapping M(GS, GS )= {(0, 0), (1, 1), (2, 2), (3, 3), (4, 4), (5, 5), (6, 6), (7, 7), (8, 8)}. In the current state of the system, replicas 0, 1, 2, 4, 5, 6, 7, and 8 are available. Let the TLP have identified the diagonal, consisting of the replicas 0, 4, and 8, as the currently cheapest WQ. Under the shown mapping, none of these replicas are directly connected in the GS in Figure 2a. Therefore we have to reconnect the replicas with the minimal additional number of replicas to reestablish communication between them. Considering the currently available replicas and the given WQ, two additional replicas are needed to reestablish communication between the elements of the WQ, e.g. replicas 1 and 6. Note, that the reconnected quorum consisting of five replicas is less likely to be available than the original quorum consisting of three replicas. Given a PNT and the set of RQs and WQs, the procedure OPTIMALMAPPING, shown in Algorithm 4, finds the optimal mapping. The procedure OPTIMALMAPPING has a runtime complexity of O(N!), where N is the number of nodes in the PNT, as there are N! possible mappings for N nodes. It does not require any knowledge of the QP used to generate the RQs and WQs, nor and knowledge of the LNT used by the QP. It only requires the set of RQs and WQs created by the QP, which makes it applicable to a wide variety of QPs. The algorithm iterates all N! mappings. For each mapping, the procedure APPLYMAPPING shown in Algorithm 4 is called. In Algorithm 3 Procedures to create Mappings and map nodes 1: procedure MAPPINGS(GS) 2: return all possible mappings of the replicas of the GS 3: end procedure 4: Inputs: nodest omap = the nodes to map, mapping = the mapping to use 5: Returns: the nodes and their mappings 6: procedure MAP(nodesT omap,mapping) 7: return {(v, v ) v nodest omap (v, v ) mapping} 8: end procedure essence, APPLYMAPPINGS uses the procedure FINDSMALL- EST to find the smallest reconnecting set of mapped nodes for each of the quorums in RQs and WQs. The procedure FINDSMALLEST is described in Algorithm 2. FINDSMALLEST works by finding the element in the power set with the least nodes that still connects the nodes in nodest oconnect. The procedure ARENODESCONNECTED shown in Algorithm 1 tests whether the nodes in nodest oconnect are connected. The procedure SELECTBESTMAPPING compares two mapped two-tuple consisting of the RQs and the WQs and returns the better one depending on the chosen measurement criterion. V. EVALUATION AND DISCUSSION In this evaluation, we examine how the read and write availability of the MCS and the TLP change when they are optimally mapped to the GS in Figure 2a. Both QPs use nine replicas. The nine replicas of the TLP are arranged as shown in Figure 2b. Figure 3 and Figure 4 show the read and write availability of the MCS and the TLP in their mapped and unmapped state. Figure 5 and Figure 6 show the read and write availability of the MCS and the TLP mapped and unmapped to the PNT where p 0.8. The optimal mapping for the MCS is M(GS, GS ) = {(0, 0), (1, 1), (2, 2), (3, 3), (4, 4), (5, 5), (6, 6), (7, 7), (8, 8)}. The optimal mapping for the TLP is M(GS, GS ) = {(0, 1), (1, 2), (2, 0), (3, 6), (4, 4), (5, 3), (6, 8), (7, 5), (8, 7)}. Both QPs use the ARW as measurement criterion for the comparison of the mappings. It can be seen in Figure 6, that the TLP behaves reasonably well with the given PNT. The read operation of the TLP loses 2% availability and 101

5 Algorithm 4 Procedures to find the optimal mapping 1: Inputs: quorums = (RQs,WQs), RQs and WQs are sets of sets of nodes mapping = the mapping to use GS = the graph structure to use 2: Returns: a modified copy of quorums; where for each quorum in RQs and WQs the procedure FINDSMALLEST is called 3: procedure APPLYMAPPING(quorums,mapping,GS) 4: return ({FINDSMALLEST(MAP(q,mapping),GS) q RQs}, {FINDSMALLEST(MAP(q,mapping),GS) q WQs}) 5: end procedure 6: Inputs: QRs, QW s = a list of all read and write quorums created by the unmapped QP GS = the graph structure the QP should be mapped to 7: Returns: the best mapping i according to the user supplied mapping comparison function SelectBestMapping 8: procedure OPTIMALMAPPING(QRs,QW s,gs) 9: optmapping empty tuple 10: for all i MAPPINGS(GS) do 11: optmapping SELECTBESTMAPPING( optmapping, APPLYMAPPING((QRs, QW s),i,gs) ) 12: end for 13: return optmapping 14: end procedure its write operation loses 3% availability for the region of p>0.8. The MCS losses 6% availability for its read and write operation as shown in Figure 5 for the same region. These are significant reductions of the operation availability, considering the goal of data replication with QP is to create a highly availability service. Figure 7 shows how the cost per operation changes for the TLP. For p < 0.4 the mapped TLP has lower write costs, because overall, less WQs are available and the WQs that are available consist of few nodes. The costs per operation of the mapped MCS are identical to the original MCS. It is interesting to note that all mappings of the MCS yield the same result. This is due to the way how the RQs and the WQ are constructed by the MCS. With nine replicas, all quorums consist of a subset of five replicas. As the MCS has no preference what five replicas it should use, all quorums are chosen with the same probability. This information can be used to accelerate the analysis of the MCS mapped to any PNT. Essentially, the OPTIMALMAPPING algorithm only must test one mapping. This is different for the TLP, where the algorithm has to test all N! mappings. VI. CONCLUSION AND FUTURE WORK In this paper, we have shown that there is a divide in the cost and availability predictions between the idealized LNT used by QPs and the costs and availability predictions when these QPs are actually used on a real-world PNT. The mapping approach formalizes this idea. Additionally, this approach finds the optimal mapping for a set of RQs and WQs on any PNT. The approach allows to evaluate the read and write availability and the read and write costs of any QP mapped on any PNT. We have shown how the mapping approach can be parameterized to facilitate different kinds of QPs. Finally, we used the presented methods to analyze the MCS and TLP when they are mapped to a particular PNT. Currently, the extremely fast-growing number of mappings is the limiting factor to which extend analyses can be carried out. To mitigate this problem, we are working into two directions. The first direction is to find properties of the mapped QPs that allow us to compare less than N! mappings. An example for of doing so is the MCS where only one mapping has to be analyzed. The other approach is to use machine learning algorithms and statistical methods to approximate the optimal mapping by analyzing only a limited number of mappings. The approach presented in this paper increases the accuracy of the real-world availability and cost predictions of QPs by considering the PNTs for the analysis. The accuracy of the predictions can be increased further by also considering the availability probabilities of the edges of the GS. Currently, all replicas fail independently, but the availability of each replica is the same. Realistically, not all replicas will be available with the same probability, therefore analyzing QPs with heterogenes replica availabilities would further increase the accuracy of the availability and cost analysis. Another area of focus is to answer the question how good a particular QP can be mapped to a specific class of GSs. By class of GSs, we mean, for example, GSs with nine vertices and two to five edges per vertex. For this work, we will use statistical methods, such as statistical inference [7], and the graph structure generator presented in [8]. REFERENCES [1] Philip A. Bernstein, Vassos Hadzilacos, and Nathan Goodman. Concurrency Control and Recovery in Database Systems. ISBN Addison Wesley, Feb. 1987, p [2] Robert H. Thomas. A Majority Consensus Approach to Concurrency Control for Multiple Copy Databases. In: ACM Transactions on Database Systems 4.2 (June 1979), pp [3] Chienwen Wu and Geneva G. Belford. The Triangular Lattice Protocol: A Highly Fault Tolerant and Highly Efficient Protocol for Replicated Data. In: Proceedings of the 11th Symposium on Reliable Distributed Systems (SRDS 92). IEEE Computer Society Press, Oct [4] Keith Devlin. Fundamentals of contemporary set theory. New York: Springer-Verlag, ISBN:

6 Figure 3: The Read (R) and Write (W) Availability of the MCS with nine replicas unmapped and mapped to the GS in Figure 2a. The lines MCS R, MCS W and MCS-Mapped R, MCS-Mapped W completely overlap. Figure 6: The Read (R) and Write (W) Availability of the TLP unmapped on a GS as shown in Figure 2b, and mapped to the GS in Figure 2a with p>=0.8. Figure 4: The Read (R) and Write (W) Availability of the TLP unmapped on a GS as shown in Figure 2b, and mapped to the GS in Figure 2a. Figure 5: The Read (R) and Write (W) Availability of the MCS with nine replicas unmapped and mapped to the GS in Figure 2a with p>=0.8. The lines MCS R, MCS W and MCS-Mapped R, MCS-Mapped W completely overlap. Figure 7: The Read (R) and Write (W) Costs per Operation of the TLP when unmapped and mapped to the GS in Figure 2a in comparison with the original TLP. [5] Hans-Henning Koch. Entwurf und Bewertung von Replikationsverfahren (in German). PhD thesis. Department of Computer Science, University of Darmstadt, Germany, June [6] Gary Chartrand, Linda Lesniak, and Ping Zhang. Graphs & Digraphs, Fifth Edition. 5th. Chapman & Hall/CRC, ISBN: , [7] Kishor S. Trivedi. Probability and Statistics with Reliability, Queuing and Computer Science Applications. 2nd Edition. ISBN John Wiley & Sons, [8] Robert Schadek and Oliver Theel. A Graph Suite Generator for Real World Quorum Protocol Analysis. In: Proc. of the 2nd Argentinian National Conference on Engineer Informatics and Information Systems (CoNaIISI 14). San Luis, Argentina: Red de Carreras de Ingeniería Informática / Sistemas de Informatión (RIISIC), Nov