Performance Evaluation of NoSQL Databases

Transcription

1 Performance Evaluation of NoSQL Databases A Case Study - John Klein, Ian Gorton, Neil Ernst, Patrick Donohoe, Kim Pham, Chrisjan Matser February 2015 PABS '15: Proceedings of the 1st Workshop on Performance Analysis of Big Data Systems Submitted By - Avinashilingam Nanjappan Krishna Chaitanya Mullapudi Yuvraj Singh Kanwar Submitted to - Prof. Suneuy Kim

2 Contents Abstract Introduction EHR Case Study Project Context Specifying Requirements Select candidate DB Design and execute performance tests Evaluation Setup Test Environment Mapping the data model Create Load Test Client Define and Execute test scripts Performance and Scalability Evaluation using strong consistency Evaluation using Eventual consistency Related work Further work and conclusions Acknowledgment References

3 Abstract Each NoSQL database comes with a different software architecture and data model. So selection of a NoSQL database for any application should be done carefully depending on the application requirements and its use cases. This paper evaluates three NoSQL databases for a healthcare use case.

4 Abstract Performance evaluation and results of the three NoSQL databases are compared in this paper. Some metric results : 1. Databases were able to handle 225 to 3200 operations per second for a typical workload. 2. When strong consistency is enforced, the throughput reduced by 10-25% compared to eventual consistency.

5 Introduction Why is Commercial Off The Shelf (COTS) product selection a complex process? Need to strike a balance between the cost, the speed of the selection process and the accuracy of the selection. There is no accurate right answer but a poor selection would impart monetary losses.

6 Introduction Why is NoSQL database selection a complex process for Big data applications in particular? Not all the requirements are available in the initial stage when there is a necessity to select a NoSQL database. Wide range of functionalities and features are available across NoSQL databases. So, straightforward comparisons cannot be made. Building a prototype to test the capabilities of each NoSQL database is not practically possible as it requires a lot of physical resources. Each day, new products get released and old products get updates making the selection process more difficult.

7 Case Study Considered An Electronic Health Record (EHR) system comprising a NoSQL database for a healthcare provider. The targeted system should be stable enough to provide critical healthcare delivery for over nine million patients in more than 100 facilities across the globe. Patient data should be retained for 99 years and for every month, one terabyte of new data is added to the system.

8 This Paper Proposes... A reliable process which can be employed by organisations to select and evaluate the scalability and performance of the NoSQL databases required for their specific needs and use cases.

9 Electronic Health Record - case study The basic steps involved: 1. Project Context 2. Specifying requirements 3. Select candidate NoSQL databases 4. Design and execute performance evaluation

10 1. Project context Replace the existing thick client application across the world that accesses a central relational database Two reasons to consider NoSQL databases : 1. Primary data store for EHR system 2. Local cache at each site to improve the latency and availability

11 2. Specifying requirements Requirement elicitation to define the critical Two driving use cases for the EHR system 1. Retrieving the most recent test results for a particular patient 2. Strong consistency for all readers when a new medical test result is written for a patient

12 3. Select the candidate NoSQL databases Based on requirement elicitation, the three NoSQL databases considered were : MongoDB Cassandra Riak Ruled out Graph databases as they do not support horizontal partitioning required for customer requirements

13 4. Design and execute the performance tests Setting up the test environment: test client platform, network topology, server platform Mapping the logical model to each database s data model and loading the synthetic data Load test client performs load tests(many concurrent requests) to evaluate the performance when the request load increases. The load test is executed in different distributed configurations to measure performance and scalability Single server to 9- server instances with replication and sharding

14 Lightweight evaluation and prototyping for Big Data

15 Evaluation Setup Test Environment: The three DBs that were tested, 1. MongoDB 2.2, a document store 2. Cassandra 2.0, a column store 3. Riak 1.4, a key-value store Database Server Configurations: 1. Single Node Server 2. 9-node configuration

16 9-node configuration Data Shards: 3 nodes Replication: 2 additional groups, 3 nodes each Used mongodb s primary/secondary feature Used Cassandra s Datacenter aware distribution feature No 3X3 Distribution feature in Riak Used a flattened configuration Testing Platform Amazon EC2 Cloud M1.large instances data and log files stored on separate EBS volume Operating System: CentOS Same EC2 availability zone

17 Mapping of Data Model Prototyping: HL7 fast healthcare Interoperability resources Logical data model: FHIR patient resources and FHIR observation resources Mapping of patient record to associated test result record One to Many Synthetic data set used for testing 1 million patient record 10 million test result record

18 Load Test Client Test Client Based on : YCSB Framework YCSB has default data models, datasets and workloads which were modified and replaced test Execution capabilities allow creation of concurrent client sessions measurement framework measures the latency for each operation performed reporting framework records latency measurements separately for read and write operations extended the YCSB reporting framework to report overall throughput

19 Defining and Executing Test Scripts 80% read and 20% write operations For test scripts: read operation retrieve the five most recent observations for a single patient write operation insert a single new observation record for a single existing patient defined a write-only workload represent the daily download from a centralized primary data store defined a read-only workload represent flushing the cache back to the centralized primary data store Results were post-processed by averaging measurements across 3 runs

20 Performance and Scalability Results Evaluation using Strong Consistency Evaluation using Eventual Consistency

21 Evaluation using Strong Consistency What is Strong Consistency? Whenever a write request is complete in one of the nodes within a cluster, the changes to the data has to be propagated to other nodes immediately. Until the changes are circulated, response to any subsequent read/write requests by any of the replicas will get delayed as all the nodes are busy in keeping each other consistent. So, strong consistency causes high latency.

22 Evaluation using Strong Consistency Strong consistency in terms of MongoDB It is configured that all the writes were to be committed to the primary server and all reads should be from primary server.

23 Evaluation using Strong Consistency Strong consistency in terms of Cassandra It is configured that all writes were committed on a majority quorum at each of the three sub-clusters, while a read required a majority quorum only on the local sub-cluster Strong consistency in terms of Riak The effect was to require a majority quorum on the entire nine-node cluster for both write operations and read operations.

24 Evaluation using Strong Consistency

25 Evaluation using Strong Consistency

26 Evaluation using Strong Consistency

27 Evaluation using Strong Consistency Performance evaluation of Cassandra Cassandra provided the best overall performance, Cassandra s read-only workload performance was nearly same as a single node configuration Write-only and read/write workload performance slightly better than the single node configuration.

28 For Cassandra, the performance gains that accrue from decreased contention for disk I/O are greater than the additional work of coordinating write and read quorums across replicas and data centers. Cassandra s data center aware features allowed a larger portion of the read operations to be completed without requiring request coordination (i.e. peer-to peer proxying of the client request), compared to Riak.

29 Evaluation using Strong Consistency Performance evaluation of Riak In test runs using the write-only workload and the read/write workload, our Riak client had insufficient socket resources to execute the workload for 500 and 1000 concurrent sessions. This resource exhaustion is because of the ambiguous documentation of Riak s internal thread pool parameter which creates a pool for each client session and not a pool shared by all the client sessions

30 Evaluation using Strong Consistency Performance evaluation of MongoDB MongoDB could not perform well because of two important factors. Sharding introduced mongodb router and config nodes into the system. Performance degraded because of the request proxying by the router node. But when the number of sessions increased, rapid router saturation takes place hence the latency remains the same across

31 Evaluation using Strong Consistency Performance evaluation of MongoDB The second issue is because of the interaction between the sharding scheme in mongodb and the workload. Range based sharing scheme was used and the key was assigned incrementally. This caused all the write operations to be done in the same shard and hence decreased the performance. When hash based sharding scheme was introduced, the tests were concluded.

32 Evaluation using Strong Consistency

33 Evaluation using Strong Consistency

34 Evaluation using Eventual Consistency MongoDB is not involved in these test results as MongoDB did not warrant additional characterization of the database of our application Set of read and writes operations were performed that resulted in eventual consistency Settings were shown in the table below

35 Cassandra - comparison of Strong and Eventual Consistency

36 Contd... At 32 client sessions, cassandra showed 25% reduction in throughput moving from strong to eventual consistency Fig above shows performance of read/write performance of Cassandra

37 Riak - comparison of Strong and eventual consistency

38 Contd 10% reduction in throughput at 32 clients configuration moving from eventual to strong consistency NO results for test configuration for 500 and 1000 concurrent sessions

39 Summary Cassandra provided the best throughput performance but with highest latency for the test configurations tested here. Two factors that attribute to the above conclusion: 1. Hash-based sharding spread the request and storage load better than MongoDB 2. Cassandra s indexing allows efficient retrieval of the most recently written records, particularly compared to Riak

40 Related Work Systematic evaluation methods allow data-driven analysis Prototyping supports component evaluation Gorton describes a rigorous evaluation method for middleware platforms Benchmarking of product is generally performed by executing a specific workload against a specific dataset Wisconsin Benchmark TPC-B Benchmark These benchmarks were developed for relational models. YCSB++

41 Riak Comparison of Strong and Eventual Consistency

42 Further work and Conclusions Some system requirements may not be fully defined product documentation is surveyed to identify viable candidate technologies prototyping and measurement is performed on a small number of candidates to collect data to make the final selection Challenges: 1. Creating the test environment 2. Validating quantitative criteria

43 Thank You!