Benchmarking Database Cloud Services

Transcription

1 Benchmarking Database Cloud Services On Sep at the annual Oracle Openworld event Oracle Executive Chairman and Chief Technology Officer, Larry Ellison presented a series of benchmarks comparing the Oracle cloud and Amazon Web Services as detailed in the following Oracle press release Oracle Beats Amazon Web Services in Head-to- Head Cloud Database Comparison. This guide details how to run these cloud benchmarks with HammerDB to make your own Cloud Database Comparisons. Introduction... 1 HammerDB Cloud Database Compatibility... 1 Cloud OLTP Workload... 2 OLTP Metrics... 3 OLTP Published Results... 4 Cloud Analytic Workload... 4 Redshift Cloud Analytic Workload... 5 Under Virtual User Options choose just 1 virtual user and select Show Output Analytic Published Results Support and Questions Introduction To compare the Oracle cloud and Amazon Web Services Oracle presented a series of benchmarks detailed in the Comparison of Database Cloud Services White Paper. In this paper Oracle documents a transaction processing and analytic workload that can be used to compare different cloud solutions. These workloads are based on the original HammerDB methodology and tools and this guide explains how to put these documented workloads into practice to perform your own cloud database comparison. This guide will focus upon the database platforms tested by Oracle. Note that at a later date after original publication Oracle supplied queries specific to Amazon Redshift that were not supplied with the original publication. These updated queries have been included with HammerDB from version HammerDB Cloud Database Compatibility Amazon Aurora and Redshift are based upon the open source databases MySQL and PostgreSQL respectively. In particular the Amazon Aurora database engine is designed to be wire-compatible with MySQL 5.6 using the InnoDB storage engine.and Amazon Redshift is based on PostgreSQL HammerDB has been tested and shown to be compatible with both Aurora and Redshift. For Aurora testing select MySQL TPC-C under Benchmark options as shown in Figure 1. 1

2 . Figure 1 Aurora selection And for Redshift select PostgreSQL TPC-H under Benchmark options as shown in Figure 2. Figure 2 Redshift selection This will enable the creation of the schemas for both OLTP and Analytic testing. Note that Aurora is specifically targeted at OLTP workloads. In particular Aurora does not have the features to support analytics such as parallel query or a column store option and therefore running the analytic tests against Aurora although possible is not likely to generate the best results. Similarly Redshift is a column store implementation of PostgreSQL and running OLTP workloads against Redshift is also possible but not recommended. This guide will therefore focus upon Aurora/MySQL for OLTP and Redshift/PostgreSQL for Analytic. Existing HammerDB workloads are compatible with the Oracle database in the cloud for both OLTP and Analytic workloads. Cloud OLTP Workload The Oracle Comparison of Database Cloud Services White Paper details the HammerDB methodology for OLTP testing precisely and therefore no modifications are required to HammerDB to perform the OLTP test. It is recommended to start with the Introduction to Transactional (OLTP) Load Testing for all Databases the Oracle OLTP Load Testing Guide and MySQL OLTP Load Testing Guide to perform the OLTP comparisons. It is important to reiterate as described in the HammerDB and Oracle documentation that the workload is based upon the TPC-C specification however does not implement a full TPC-C specification in any way and the results cannot be compared with official TPC-C publications. 2

3 Oracle specify 2 additional schema/system for the OLTP test: 1. The database must be encrypted 2. The system must preserve all committed transactions and ensure consistency in the event of the failure of 2 storage devices For Aurora/MySQL/InnoDB the storage engine must have the following setting innodb_flush_log_at_trx_commit=1 and Oracle operate in the default redo log setting. Emphasis is likely to be placed on the ability of the redo logs to sustain a high throughput of transactions. OLTP Metrics It is important to reiterate as described in the HammerDB and Oracle documentation that the workload is based upon the TPC-C specification however does not implement a full TPC-C specification in any way and the results cannot be compared with official TPC-C publications. Oracle specify that the metric is OLTP Transactions per Minute, which includes all transaction not just New Orders per Minute and therefore some clarification is beneficial. When running the HammerDB online transaction counter you will see an approximation of the database internal metric for transactions per minute sampled across the given time interval and extrapolated to a minute (shorter time intervals are likely to overestimate the transaction rate). Figure 3 illustrates and example of Oracle TPM. Figure 3 Oracle TPM TPM is recorded and used as a metric for monitoring the test as it is the most lightweight metric from database in memory tables without impacting the schema being tested. The metrics used can be viewed at a command prompt with the following statements. Oracle select sum(value) from gv$sysstat where name = 'user commits' or name = 'user rollbacks' This value is also shown in the Load Profile section of an Oracle AWR report as transactions per second and also displayed in Oracle Enterprise Manager. TPM will capture all database activity if work in addition to the HammerDB workload is taking place simultaneously. 3

4 MySQL/Aurora show global status where Variable_name = 'Handler_commit' or Variable_name = 'Handler_rollback' It is absolutely vital to note that the TPM value records different metrics across different databases and therefore TPM can be used to compare the same database between systems however cannot be used to compare between different databases. At the end of a HammerDB timed test the TPM value is reported and is also accompanied by an additional value termed NOPM. NOPM records the New Orders Per Minute and in this case is a database independent metric and is selected directly from the schema under test. For precisely this reason NOPM is not displayed during a test and is only sampled both before and at the end of a timed test interval. Therefore to provide a comparison between different databases a timed test should be performed and the NOPM value recorded and reported. OLTP Published Results For the OLTP test Oracle specifies that customers are free to run this benchmark, but Oracle must certify all published results. This holds true for publishing results of all database benchmarks involving the Oracle database due to license conditions (and databases with commercial licenses such as Microsoft SQL Server and DB2 also have similar clauses) however you are free to publish benchmarks against open source databases without seeking permission from any vendor or authority. Cloud Analytic Workload In similarity to Cloud OLTP workload the Oracle Comparison of Database Cloud Services White Paper details the HammerDB methodology for analytic testing however in this case with modifications. Nevertheless given the similarity of the workloads it is recommended to start with the Introduction to DSS (Query and Analytics) Load Testing for all Databases and the Oracle DSS/Data Warehousing Testing Guide. It is again important to reiterate as described in the HammerDB and Oracle documentation that the workload is based upon the TPC-H specification however does not implement a full TPC-H specification in any way and the results cannot be compared with official TPC-H publications. For the Cloud Analytic workload the database is created and loaded according to the TPC-H specification HammerDB by default can create and load schemas of scale factor 1, 10, 30, 100, 300 and 1000 which corresponds to 1GB, 10GB, 30GB, 100GB, 300GB and 1TB respectively. HammerDB will create the schema and load the data. The time this takes is dependent on multiple factors in particular storage performance is a key factor for inserting a large number of rows into the database as each insert requires redo logging and there is the potential for contention as the number of virtual users increases all inserting rows into the same table at the same time. For this reason especially as the Oracle specification requires a schema size of 10TB, it is recommended to create the schema with HammerDB using the Generating and Bulk Loading Data feature and this guide details how to do this for both Oracle and Redshift and this is particularly recommended when uploading data to the cloud. You are permitted to run both the in-built TPC-H query workload and the Oracle provided query set against any size schema however the Oracle workload provides 13 new analytic queries that is run in sequence as per the original power test of TPC-H. This new query set compatible with Oracle, Redshift/PostgreSQL and MySQL/Aurora is enabled under the TPC-H Driver Script Options dialog by selecting the Cloud Analytic Queries checkbox. This query set reports the geometric mean of the completed queries that returns rows for circumstances where the query set is run on a scale factor size of less than 10TB. Given the similarity of the Oracle implementation to the existing TPC-H workload the following example illustrates running the workload against Amazon Redshift. 4

5 Redshift Cloud Analytic Workload Ensure that your Redshift cluster is active and note your Endpoint name given above the cluster properties. Figure 4 Redshift Console Also ensure that access is enabled to the cluster both by defining a user and a security group and allowing access through your firewall. 5

6 Figure 5 Create Security Group If you have not already done so create the schema within Redshift using the HammerDB Generating and Bulk Loading Data feature. To run the Cloud Analytic Workload with HammerDB on your host select PostgreSQL and TPC-H under Benchmark Options. Figure 6 Benchmark Options Under PostgreSQL TPC-H Driver Options use the Redshift Endpoint as your PostgreSQL host and 5439 as your port. Set the user and password to the credentials you have set under the Amazon AWS console. Finally select the Cloud Analytic Queries and Redshift Compatible Checkbox. 6

7 Figure 7 Driver Options Select Load under the Driver Script to load the script as shown in Figure 8. Figure 8 Driver Script Loaded 7

8 Under Virtual User Options choose just 1 virtual user and select Show Output. Figure 9 Virtual User Options HammerDB will show a single Virtual User created with waiting status as shown in Figure 10. Figure 10 Virtual User Created Run the Virtual User and HammerDB will execute the Cloud Analytic Queries against your Redshift database. 8

9 Figure 11 Virtual User Running Once all queries are completed the Geometric Mean of all queries is reported as shown in Figure 12. 9

10 Figure 23 Geometric Mean Note that when running the queries against data sets smaller than the specified 10TB this may result in some queries not returning rows, therefore for your calculations HammerDB calculates the geometric mean only of queries that returned rows. Analytic Published Results As per the OLTP test also for the Analytic test Oracle specifies that customers are free to run this benchmark, but Oracle must certify all published results. This holds true for publishing results of all database benchmarks involving the Oracle database due to license conditions (and databases with commercial licenses such as Microsoft SQL Server and DB2 also have similar clauses) however you are free to publish benchmarks against open source databases without seeking permission from any vendor or authority. Support and Questions For help use the HammerDB Sourceforge forum available at the HammerDB sourceforge project. 10