Modelling Replication in NoSQL Datastores

Transcription

1 Modelling Replication in NoSQL Datastores Rasha Osman 1 and Pietro Piazzolla 1 Department of Computing, Imperial College London London SW7 AZ, UK rosman@imperial.ac.uk Dip. di Elettronica e Informazione, Politecnico di Milano, via Ponzio 3/5, 0133 Milano, Italy piazzolla@elet.polimi.it Abstract. Distributed NoSQL datastores have been developed to cater for the usage scenarios of Web.0 applications. These systems provide high availability through the replication of data across different machines and data centers. The performance characteristics of NoSQL datastores are determined by the degree of data replication and the consistency guarantees required by the application. This paper presents a novel performance study of the Cassandra NoSQL datastore deployed on the Amazon EC cloud platform. We show that a queueing Petri net model can scale to represent the characteristics of read workloads for different replication strategies and cluster sizes. We benchmark one Cassandra node and predict response times and throughput for these configurations. We study the relationship between cluster size and consistency guarantees on cluster performance and identify the effect that node capacity and configuration has on the overall performance of the cluster. 1 Introduction Scalable NoSQL datastores have been developed for simple read/write operations that mainly characterize distributed Web.0 applications [0]. They have been designed for horizontal scalability and high availability and thus are more suitable for elastic cloud deployment than traditional databases. High availably is achieved through the replication of data across different machines and different data centers. Replication of data is asynchronous, in which updates/writes are written to replicas in the background allowing for faster response times, albeit with a weaker guarantee of data consistency [] in comparison to traditional databases. Current web applications and services require low response times, with high latency having a direct impact on revenue for large service providers [13, 1, 19]. The trade-off between response time and consistency in NoSQL data stores means that providers need to be able to strike a balance between these two factors. To improve system deployment and user satisfaction performance predication techniques are needed to estimate NoSQL datastore response times given a consistency level guarantee. Performance prediction techniques would give service

2 providers the ability to test the resilience, scalability and latency of a configuration for different workloads and hardware environments. With pay for use cloud hosting, providers would be able to deploy the correct amount of servers for their configurations to meet service level agreements. Performance modelling has mostly concentrated on traditional databases [18] with queueing networks being the main modelling formalism. Other forms of quorum systems have been modelled in [1, 1]. In this paper, we present a novel performance modelling study of the Cassandra [1] NoSQL datastore using queueing Petri nets (QPNs) []. Queueing Petri nets have been applied in the performance evaluation of distributed systems [10, 9, 11], grid environments [15] and relational database concurrency control [17, ]. This work contributes a simple QPN model that is parameterized by benchmarking one Cassandra node. We show that the model can scale to represent the performance of read workloads for different replication strategies and cluster sizes and predict response times and throughput for these configurations. The relationship between cluster size and consistency guarantees is identified for scaled and non-scaled workloads. In addition, we identify the effect of node capacity on the overall performance of the cluster. The rest of this paper is organized as follows. Section gives an introduction to replication in Cassandra. Section 3 presents the specification of the QPN model for a Cassandra cluster. Experimental results and analysis are in Section. Finally, Section 5 concludes the paper and provides directions for future work. Replication in Cassandra Apache Cassandra is a distributed extensible column store [0, ] originally developed at Facebook [1] to provide high scalability and availability with no single point of failure. To guarantee even distribution of data across the cluster, the key value space is mapped onto a ring. The ring is divided into ranges, in which each node is assigned one or more random ranges on the ring. A node can only store keys that fall within its assigned range and thus the node becomes the coordinator node for these key values. Cassandra supports multi-master replication with configurable quorum style read and write consistency [0, ]. Multi-master consistency means that a client can contact any node, irrespective of whether it stores/replicates the requested key. For a cluster of N nodes, the replication factor (RF) is the number of times a key is replicated across the cluster. Each coordinator node is in charge of storing its data locally and replicating it to RF 1 nodes. Hence, a node will store RF/N% of the key ranges. The consistency level (CL) is the number of replicas in which the read/write operation must succeed before returning a success to the client. The effect of the consistency level is different for reads and writes [7]. For the purpose of this paper, we will explain replication for read operations. Each node in a Cassandra cluster has local information about the key ranges stored at other nodes. For read requests, a coordinator node will forward requests to CL of the RF replicas of a key for requests not stored locally and to CL 1

3 3 replicas for locally stored requests. The coordinator forwards the key requests to the replica nodes that are responding the fastest. Figure 1 shows a cluster with six nodes, a replication factor of 3 and a consistency level of. A client contacts node 5 with a read request for key A, which is replicated on nodes 1, & 3. Therefore, node 5 will act as the coordinator node for the client and send a read request for key A to of the 3 replicas, i.e., the configured consistency level. The client s request will be blocked until both of the contacted replicas reply with their locally stored values for key A. Node 5 will compare the returned values and relay the most recent value of key A to the client. 1 A A client 5 3 A Fig. 1. Example of a read operation on a node Cassandra cluster with replication factor 3 and consistency level. 3 QPN Model of a Cassandra Node To model a node in a Cassandra cluster, we need to represent (1) the scheduling of requests processed by a node, () the synchronization and blocking of requests at the coordinator node that need to be fulfilled by remote nodes and (3) the routing of replica requests based on the replication factor and consistency level. To model scheduling and processing of requests, a queueing station is the most suitable formalism. For synchronization between nodes and blocking of requests Petri nets are more suitable. Our modelling approach is based on queueing Petri nets [], an extension of coloured stochastic Petri nets, which provide a natural formalism in combining scheduling and synchronization in one model. In queueing Petri nets, queueing places consist of two components: the queue, and the depository. Tokens enter the queueing place through the firing of input transitions, as in other Petri nets; however, as the entry place is a queue, they are placed in the queue according to the scheduling strategy of the queue s server. When a token has been serviced it is placed in the depository where it can be used in further transitions. Timed queueing places have variable scheduling

4 strategies and service distributions; while immediate queueing places impose a scheduling discipline on arriving tokens without a service delay. Due to space limitations, we refer the reader to [] for a more detailed description of QPNs. Figure represents the QPN model of a Cassandra cluster showing two nodes. The bordered area highlights the parts of the model representing one node. We refer to an arbitrary node in the cluster by node i. As all nodes are similar in the cluster, we will describe our QPN model by referring to node i. The token colours used in the model are summarized in Table 1. Fig.. A queueing Petri Net model of a Cassandra cluster. In the QPN model of Figure, the clients are represented by a timed queueing place with an infinite-server queue and an exponentially distributed think time. The client place is initialized with the number of client read requests sent to the cluster. Read requests are represented by read tokens. The enter-cluster immediate transition distributes the incoming requests randomly between the different nodes in the cluster by depositing read tokens in the enter-node i immediate queueing place of each node. This ensures that all nodes are equal in receiving requests, similar to Cassandra s multi-master architecture. The default configuration of Cassandra restricts the number of concurrent read operations to 3 [7]. This is modelled using the ordinary place, threads i, which is initialized to 3 tokens. Incoming requests queue in FIFO order in the enter-node i place and the immediate transition process-requests i fires, providing

5 5 Table 1. Token Colours. Token Color Description Place T thread token (initialized to 3) threads i local-read local incoming read request node i read remote read request enter-node i & node i blocked reads blocked i client incoming and returning requests client & exit-cluster read i forwarded read request from node i enter-node j & node j, where j i R replication responses between nodes replica-response i access to the processor when there is at least one token in the threads i place. The immediate transition process-requests i determines the proportion of local and remote requests entering node i based on the replication factor. This is discussed below. The processing at node i is represented by node i, a timed queueing place, representing an /M/1 P S queue. Non-Local Requests. A non-local or remote request is a request for a key that is not stored at node i. The probability of a non-local request entering node i is equal to 1 RF/N. When the immediate transition process-requests i fires producing a non-local request, it removes a read token from the enter-node i place and deposits a read token in the node i timed queueing place. For simplicity, we assume that the mean service time of a remote read request is similar to that of a local read request. However, we note that a remote request does not use thread tokens, thus not affecting a node s ability to accept local reads. After processing, a remote request is blocked awaiting replies from replica nodes. To synchronize replica requests between the coordinator node i and replica nodes, node i blocks remote read requests awaiting replies by firing the immediate transition distribute-requests i. This is executed by placing a read token in the blocked i ordinary place and a token of color read i in the enter-node j place of node j, where j i & the total number of contacted nodes is equal to CL. The choice of node j depends on the consistency level and replication factor (discussed in Section 3.1). The color read i distinguishes forwarded replica requests by node index so that the reply can be returned back to the correct coordinator. Any node j receiving a token read i in the enter-node j place will process it similarly to a local-request (see below). When a read i token leaves node j the immediate transition distribute-requests j will return a thread token to the threads j place and notify node i of the arrival of the request by depositing a token in the corresponding replica-response i place of node i. This will enable the unblock i immediate transition, which will fire when the number of tokens in the replicaresponse i place is equal to the consistency level, i.e., the required nodes have responded. One read token (representing the blocked read request) is removed from the blocked i place and CL tokens are removed from the replica-response i

6 place and one read token is deposited in the exit-cluster place. The read request is then transferred back to the client. Local Requests. A local request is a request for a key that is stored at node i. The probability of a local request entering node i is equal to RF/N. For a local request, when the immediate transition process-requests i is enabled, it will remove a read token from the enter-node i place and a thread token from the threads i place and deposit a local-read token in the node i timed queueing place. In addition, a read token is deposited in the blocked i place so that the functionality of the release of a blocked read request is the same for both local and remote requests when CL > 1. After processing the local request, the distribute-requests i transition is enabled and the thread token is returned and one token is deposited in the replicaresponse i place, representing the completion of one request. If CL > 1, then the distribute-requests i transition will deposit a read i token in the enter-node j place of CL 1 replica nodes. The contacted replica nodes will process the read i tokens in the same manner described in the previous section for non-local requests. The unblock i transition will fire when the number of tokens in the replicaresponse i place is equal to the consistency level, i.e. all the required nodes have responded (this includes the token deposited by node i ). In the case when CL = 1, the unblock i transition will be immediately enabled. Similar to non-local requests, one read token (representing the blocked read request) is removed from blocked i and CL number of tokens is removed from the replica-response i place and one read token is deposited in the exit-cluster place. Finally, the read request is transferred back to the client. 3.1 Forwarding to Replicas To model replica forwarding, we assume that the coordinator contacts the replica nodes randomly. The probability of a replica node j receiving a request from node i is modelled as the probability of the firing of the arc connecting the distributerequest i immediate transition with the enter-node j place of node j. We describe this below. Figure 3 illustrates a token ring for an N node cluster. Assume that a node s index represents its position on the virtual range ring and hence the key space is divided into N ranges. A node will replicate its assigned keys by moving clockwise on the ring. 1 Therefore, when RF > 1, node i replicates its key range on the subsequent RF 1 nodes on the ring, i.e. nodes of indices: S = [(i mod N) + 1, ((i + RF ) mod N) + 1]. (1) In addition, node i is a replica node for the preceding RF 1 nodes, i.e., it stores a replica for the key values of nodes: 1 This is the default method used by Cassandra1..

7 7 P = [((i RF ) mod N) + 1, ((i ) mod N) + 1]. () When node i receives a local key request and CL > 1, it randomly chooses a key value, k, from the local key range: L = P {i} = [ ((i RF ) mod N) + 1, i ]. (3) Node i will identify the RF 1 replica nodes of k and randomly choose CL 1 nodes in which to forward a replica key request. The number of random node combinations is ( RF CL 1), thus forming a collection of sets, Fl, where l = 1,..., ( RF CL 1), and for any nodej F l, j L & j i. Each set in F l represents a set of connections/arcs between the distribute-request i immediate transition of node i and the enter-node j place of all nodes in F l. When the distributerequest i transition is enabled it will randomly fire one set of connections/arcs and CL 1 downstream nodes will receive tokens simultaneously. Algorithm 1 details the method for identifying the connections between the distribute-request i immediate transition and the enter-node j places. Figure illustrates an example of the different combinations for the incidence function of the distribute-request i immediate transition when RF = and CL = 3. In general, if CL = RF, then node i will send the request to all replica nodes and await all their replies. If CL =, it will choose one node randomly. Request for keys not stored on node i will be key ranges assigned to the remaining nodes on the ring. Assume the set of all N nodes in the ring is A, then the remaining nodes on the ring are: O = A \ L = A \ [ ((i RF ) mod N) + 1, i ]. () When node i receives a remote key request, it randomly chooses a key value from the nonlocal key range O and identifies its RF replica nodes, then randomly chooses CL nodes to forward a replica request. Similar to local requests, the number of random node combinations is ( RF CL), which represents the number of connection/arc sets between the distribute-request i immediate transition of node i and the enter-node j places, where j O & j i. Algorithm describes the method for identifying the nonlocal replica forwarding connections between the distribute-request i immediate transition and the enter-node j places. In general, if CL = RF, then node i will send the request to all replica nodes and await all their replies. If CL = 1, it will choose one node randomly. Experimental Results.1 Experimental Setup For our experiments, we use Cassandra version 1., Datastax distribution [7]. The loading and benchmarking of the Cassandra cluster was conducted using the Cassandra-stress tool [3], a Java-based stress testing utility included with the Cassandra Datastax distribution. The nodes of the Cassandra cluster used

8 8 Fig. 3. A token ring of a cluster of N nodes. S represents the replica nodes for node i s assigned key range. P represents the RF 1 predecessors to node i and O are the coordinator nodes for key ranges not stored on node i. enter_node R node i enter_node R3 distribute-request i enter_node R Fig.. The combination of connections for the distribute-request i transition when RF = and CL = 3 on processing a local read request at node i. For each firing one connection set is chosen randomly.

9 9 Algorithm 1 Replica Request Forwarding: Local Requests Require: N: cluster size, RF : replication factor, CL: consistency level where RF > 1 & 1 < CL RF & i: index of current node 1: key begin = ((i RF ) mod N) + 1) : key end = i 3: for k = key begin key end do define local key range calculate RF 1 replica indices, excluding node i : replica range = {k} [(k mod N) + 1, ((k + RF ) mod N) + 1] \ {i} 5: Define array of replica sets replica sets[counter, nodes[ ] ] calculate and store the ( ) replica range CL 1 combinations : replica sets[counter][ ] = the CL 1 combinations of nodes of replica range 7: for j = 1 replica sets.counter do iterate over all combinations 8: for l = 1 replica sets[j].nodes[ ] do iterate over the nodes of combination j 9: Create connection between distribute-request i transition 10: and entry node place of replica sets[j].nodes[l] 11: end for 1: end for 13: end for connections are enabled when a local read token leaves node i Algorithm Replica Request Forwarding: NonLocal Request Require: N: cluster size, RF : replication factor, CL: consistency level where RF > 1 & 1 CL RF, A: set of the indices of the N nodes & i: index of current node 1: Let O = A \ [(i RF ) mod N) + 1, i ] : key begin = min{j O}; key end = max{j O} 3: for k = key begin key end do define nonlocal key range calculate RF replica indices : replica range = {k} [(k mod N) + 1, ((k + RF ) mod N) + 1] 5: Define array of replica sets replica sets[counter, nodes[ ] ] calculate and store the ( ) replica range CL combinations : replica sets[counter][ ] = the CL combinations of nodes of replica range 7: for j = 1 replica sets.counter do iterate over all combinations 8: for l = 1 replica sets[j].nodes[ ] do iterate over the nodes of combination j 9: Create connection between distribute-request i transition 10: and entry node place of replica sets[j].nodes[l] 11: end for 1: end for 13: end for connections are enabled when a read token leaves node i

10 10 in the experiments were hosted on the Amazon EC cloud environment [1]. More specifically, each machine hosting a Cassandra node was an instance of the m1.xlarge 3 AMI type. Since we were not investigating the impact of multicore processing we disabled one of the virtual cores of the instance. Hence, each node behaved as a single core CPU with all the available cache capacity at its disposal. Cassandra 1. was set up using the default configuration, except for the read repair chance value that was set to 0 in order to avoid possible interferences by the random automatic updating of data across the cluster. The stress tool used the default configurations, which included the default keyspace options. To control the number of requests entering the cluster we modified the default number of threads of the stress tool to represent different request rates. The client machine hosting the Cassandra-stress tool was external to the cluster (similar to Figure 1) and hosted on the Amazon EC cloud environment using the same configurations as the Cassandra nodes, except with both cores active to avoid becoming the bottleneck in the system. The stress tool randomly sent requests to each of the nodes on the cluster. All the experiments discussed in the following Section focused on read replication. For each experiment, the Cassandra cluster had a specific configuration in terms of number of nodes, replication factor and consistency level. This was controlled by the Cassandra-stress tool, in which each experiment specified the replication factor and the consistency level (see [3] for details). During each test two million keys were written to the cluster, and then separate runs would read these keys while varying the number of concurrent threads used by the stress tool for each run. The QPN models in this paper were developed and solved using QPME.0 [8]. QPME.0 (Queueing Petri net Modeling Environment) is an open source performance modelling tool based on the QPN modelling formalism. QPME.0 is composed of two components, a QPN editor (QPE) and a simulator for QPNs (SimQPN). All our simulation runs used the method of non-overlapping batch means (with the default settings) to estimate the steady state mean token residence times with 95% confidence intervals. To parameterize the queueing Petri net models, we setup a Cassandra cluster with one node and ran the Cassandra-stress tool with concurrent threads ranging from 1 to 0 threads. The number of concurrent threads run by the stress tool corresponds to the number of jobs entering the system. We noticed that the throughput of the node stabilized at 5 and more threads, with an average throughput of.37 requests per ms. We used the mean request response time for five threads to approximate the response time for one thread, and thus the service rate of one read request. We have found the service rate of a read request to be. requests/ms. To calculate the client (thread) think time, we used the 3 Specification of a m1.xlarge instance (as of January 01): ECU, cores, 7. GB RAM, 8GB HD. The cores of physical processors used by Amazon for the m1.xlarge instance share the L1 and L caches.

11 11 utilization law. We measured the mean utilization for the client when running the stress tool, in addition to the mean throughput arriving at the client. From that we calculated the mean client think time as 0.07 ms.. Results As described in the previous Section, the Cassandra-stress tool was deployed on a client node and a set of experiments on different sized Cassandra clusters was conducted. Each experiment had n nodes, RF replication factor and CL consistency level, where n = to, RF = 1 to n and CL = 1 to RF. We refer to each configuration using a triple: (a-b-c), where a represents the number of nodes, b represents the replication factor and c represents the consistency level. This convention will be used throughout to discuss our results. Figure 5 shows the measured and predicted mean response times and mean throughput for a node cluster for different RF and CL configurations. The results for clusters of size and 3 exhibit similar behaviour as that of a node cluster and have been omitted due to space restrictions. Measured and predicted mean response times for 1,, 3 and node clusters for 0, 0 and 0 threads are shown in Table. Response time and throughput. From Figure 5 and Table, when the cluster size is fixed, in this case nodes, the read request response time increases linearly as the workload intensity increases. This linear increase within a fixed RF and CL is due to the increase in job queueing time as jobs wait for free processing threads at each node. This effect is evident in the mean throughput of the cluster, which increases and then reaches its maximum at about 0 jobs/threads for the node cluster. Increasing the RF and CL within a fixed cluster size, decreases the performance of the system. This is due to increased inter-node traffic due to increased replica request forwarding between nodes, thus leading to a sharper linear increase as shown in Figure 5. The QPN model accurately predicts the linear increase in response times for all cluster sizes and configurations, as it accurately reflects the blocking, queueing and processing of the requests. However, the model generally underestimates the response time as the model does not represent the effect of network delay and the increased synchronization processing between the nodes which will increase response times and lower throughput. The QPN model gives excellent predictions for the node configurations. For the 3 node and node configurations, the predictions have an average error ranging from % to 0% for the majority of configurations for both mean response times and throughput. However, for the (-3-1) & (--) configurations the mean prediction error for response time and throughput are 5% and 30% respectively. For (3-3-1) configuration the mean error for response time and throughput is 30% & 8%, for (3-3-) it is 35% & 35% and for (--1) it is 0% & 0%. When conducting the experiments we noticed that as the replication factor approached the cluster size, the stability of the cluster and its performance was affected. This can be due to the high synchronization between the nodes, increasing the inter-node

12 1 traffic as the consistency level increased. In addition, as the clusters are hosted on virtual machines, it is probable that the increased incoming requests to each node affected the ability of the virtual processors. Moreover, the instability of the node cluster was evident with high variability in comparison to the and 3 node clusters. The accuracy of the model was not affected by this instability at higher consistency levels. The increased inter-node traffic increased request queueing mean response time mean throughput Measured Predicted 1 Mean Response Time (msec) Measured --1 Predicted -- Measured -- Predicted Throughtput (jobs/msec) Measured -1-1 Predicted --1 Measured --1 Predicted -- Measured -- Predicted Number of Threads Number of Threads (-1-1) & (--x) Measured -3-1 Predicted 18 1 Mean Response Time (msec) 1-3- Measured -3- Predicted Measured Predicted Number of Threads Throughtput (jobs/msec) (-3-x) Measured -3-1 Predicted -3- Measured -3- Predicted -3-3 Measured -3-3 Predicted Number of Threads Measured Predicted 1 Mean Response Time (msec) Measured -- Predicted --3 Measured --3 Predicted -- Measured -- Predicted Throughtput (jobs/msec) Measured -- Measured --3 Measured -- Measured --1 Predicted -- Predicted --3 Predicted -- Predicted Number of Threads Number of Threads (--x) Fig. 5. Measured and predicted mean response times and throughput for a node cluster for different configurations.

13 Table. Measured and predicted mean response times for different configurations and cluster sizes for 0, 0, & 0 thread workloads. 1 node nodes 3 nodes nodes RF threads CL Meas. Pred. Meas. Pred. Meas. Pred. Meas. Pred. Meas. Pred. Meas. Pred. Meas. Pred. Meas. Pred. Meas. Pred. Meas. Pred

14 1 time for processing threads; which becomes more dominant in overall response times in comparison to other factors. Previous studies have shown that CPU intensive workloads on the Amazon EC environment have unstable response times with up to 5% variance [5]. Within this range, the model predictions are reliable for capacity planning, giving the upper bounds on performance and throughput of the clusters based on benchmarking the performance of one node. This is clear from the absolute difference between measured and predicted values presented in Table. Replication and consistency. Higher replication levels increase the likelihood of faster request responses. This is closely tied to the consistency level. From the results (and intuitively), given the same consistency level and cluster size, increasing the replication factor gives better performance. In contrast, for the same cluster size and replication factor, increasing the consistency level will worsen performance due to the increased waiting time for the extra requests to be routed back to the original node. Both hold irrespective of workload (e.g. Figure 5) and cluster size. When increasing the cluster size, the replication factor and consistency level dictate the amount of inter-node traffic, therefore a larger cluster size with more replication and a higher consistency level may mean longer response times. Comparing different configurations, given that the replication factor and consistency level are fixed, performance improves with the increase in cluster size when job queueing time starts to dominate response time. Thus, we see improved performance with the increase in cluster size and a linear increase in throughput. From Table, this holds for workloads of 30 and more threads (this holds for the predicted results irrespective of workload), e.g., at 0 threads/jobs (--1) exhibits better performance than (3--1) and (--1). However, we notice that this does not hold for (3--x) in comparison to (--x), as the node cluster has the advantage of full replication. Our original experiments considered workloads that do not scale with cluster size. To mitigate the differences in request queueing between clusters of different sizes, we investigated the effect of replication and consistency level when scaling the workload by cluster size. Figure gives one example of the effect of scaled workload on the mean response times and throughput for number of threads (jobs) t: when t = n 10 and t = n 0, in comparison to the non-scaled workload t = 0 for (n-1-1) configurations. For a fixed consistency level and replication factor, increasing the cluster size increases response time in comparison to the non-scaled workload. However, the throughput level is sustained for the scaled and non-scaled workloads and increases linearly with cluster size. In this scenario, all nodes experience the same number of incoming jobs irrespective of cluster size. Therefore, increasing the cluster size increases the detrimental effect on performance of inter-node communication, synchronization and request blocking. These aspects have minimal impact on throughput, as the number of jobs processed at each node is bounded by the number of available processing threads, irrespective of cluster size.

15 Mean Response Time (msec) t = 10xnodes -measured t = 0xnodes -measured t = 0 -measured Throughtput (job/msec) 8 t = 10xnodes -measured t = 0xnodes -measured t = 0 -measured 1 t = 10xnodes -predicted t = 0xnodes -predicted t = 0 -predicted t = 10xnodes -predicted t = 0xnodes -predicted t = 0 -predicted Cluster Size (a) Cluster Size (b) Fig.. Measured and predicted mean (a) respone times and (b) throughput for (n-1-1) configurations comparing scaled and non-scaled workloads. 5 Conclusions and Future Work This paper presents a generic QPN model of a Cassandra multi-master cluster under different read workload intensities and consistency configurations. Given the performance measures of one Cassandra node the model was able to predict response times and throughput for different configurations for Cassandra clusters running on the Amazon EC cloud platform. This work detailed the relationship between cluster size, replication factor and consistency level on response time and throughput for scaled and non-scaled workloads. Moreover, our experiments identified the effect that node capacity and configuration has on the overall performance of the cluster. The model presented can be applied to other NoSQL datastores that share a similar replication model with Cassandra. The results presented are based on random access to keys; skewed access can be modelled by modifying Algorithms 1 & through setting the connection probabilities of different node set combinations. We plan to extend the QPN model to model write and mixed workloads and explicitly represent network delays. Furthermore, as the QPN simulation time increases with cluster size, we will investigate deriving analytical formulas based on the cluster size, replication factor, consistency level and node capacity to determine the trade-off between different configurations. This would require further experimentation with larger clusters and increased runs to mitigate the effect of virtualization. Acknowledgments. The authors would like to thank the anonymous reviewers for their comments that helped improve the quality of this paper. This work has been partially supported by the AWS in Education research grant from Amazon and by the ForgeSDK project sponsored by Reply S.R.L. References 1. Amazon Web Services - Elastic Cloud Computing.

16 1. Bause, F.: Queueing Petri Nets A Formalism for the Combined Qualitative and Quantitative Analysis of Systems. In: Fifth Intl Workshop Petri Nets and Performance Models (1993) 3. Cassandra-stress benchmarking tool. cassandra/1./cassandra/tools/toolscstress_t.html. Cattell, R.: Scalable SQL and NoSQL Data Stores. SIGMOD Rec. 39,, 1 7 (011) 5. Cerotti, D., Gribaudo, M., Piazzolla, P., Serazzi, G.: End-to-End Performance of Multi-core Systems in Cloud Environments. EPEW 013, LNCS vol. 818, 1 35 (013). Coulden, D., Osman, R., Knottenbelt, W.J.: Performance Modelling of Database Contention using Queueing Petri Nets. In: th ACM/SPEC International Conference on Performance Engineeering (013) 7. Datastax Documentation Apache Cassandra (01) 8. Kounev, S., Spinner, S., Meier, P.: QPME.0 - A Tool for Stochastic Modeling and Analysis Using Queueing Petri Nets. In From Active Data Management to Event-Based Systems and More, LNCS vol., (010) 9. Kounev, S.: Performance Modeling and Evaluation of Distributed Component-based Systems using Queueing Petri Nets. IEEE Trans. Software Engineering, 3, 7, 8 50 (00) 10. Kounev, S., Buchmann, A.: Performance Modelling of Distributed e-business Applications using Queuing Petri Nets. In Proc. IEEE International Symposium on Performance Analysis of Systems and Software, (003) 11. Kounev, S. et al.: Automated Simulation-Based Capacity Planning for Enterprise Data Fabrics. In Proc. th SIMUTOOLS 011, 7 3 (011) 1. Lakshman, A., Malik, P.: Cassandra: a Decentralized Structured Storage System. SIGOPS Operating Systems Review,,, 35 0 (010) 13. Linden, G.: Make Data Useful. StanfordDataMining pdf (00) 1. Linden, G., Mayer, M.: Presented at Web /11/marissa-mayer-at-web-0.htm (00) 15. Nou, R., Kounev, S., Julia, F. and Torres, J.: Autonomic QoS Control in Enterprise Grid Environments using Online Simulation. Journal of Systems and Software, 8, 3, 8 50 (009) 1. Omari, T., Derisavi, S., Franks, G. and Woodside, M.: Performance Modeling of a Quorum Pattern in Layered Service Systems. In: QEST 007 (007) 17. Osman, R., Coulden, D., Knottenbelt, W. J.: Performance Modelling of Concurrency Control Schemes for Relational Databases. In: 0th International Conference on Analytical and Stochastic Modelling Techniques and Applications, LNCS vol. 798, (013) 18. Osman, R., Knottenbelt, W.J.: Database System Performance Evaluation Models: A Survey. Performance Evaluation. 9, 10, (01) 19. Schurman, E., Brutlag., J.: Performance Related Changes and their User Impact. Presented at Velocity Web Performance and Operations Conference (009) 0. Stonebraker, M., Cattell, R.: 10 Rules for Scalable Performance in Simple Operation Datastores. Commun. ACM 5,, 7 80 (011) 1. Tadj, L., Rikli, N.E.: Matrix Analytic Solution to a Quorum Queueing System. Mathematical and Computer Modelling. 3, 3, (000)