35 Database benchmarking 25/10/17 12:11 AM. Database benchmarking

Transcription

1 Database benchmarking 1

2 Database benchmark? What is it? A database benchmark is a sample database and a group of database applications able to run on several different database systems in order to measure performance each system and relate it to price of each system (price/ performance ratio) Performance metric is typically a throughput metric (work/second, e.g number of transactions executed per second, number of queries executed per hour) Price metric is typically five years cost of ownership 2

3 Domain specific benchmarks A domain specific benchmark specifies a synthetic workload characterizing typical applications in the problem domain For example: numeric computations benchmark (number of floating point operations per second), transaction processing benchmark (number of transactions per second), query processing benchmark (number of queries per hour) 3

4 Properties Relevance A benchmark should be meaningful with a target domain Clarity A benchmark should clearly determine the measured characteristics Scalability It should be possible to scale up a benchmark Linearity, orthogonality, and monotonicity Linear scaling up a benchmark linearly increases its complexity benchmark parameters should be independent on each other, scaling benchmark up should always make it more demanding 4

5 Properties Coverage A benchmark should not oversimplify a typical environment Acceptance A benchmark should be accepted by the vendors and users 5

6 TPC-C benchmark (1992) Constructed by Transaction Processing Council (de facto industry standards body for online transaction processing performance, TPC-C models a wholesale supplier managing orders Transaction types: New-order: enter a new order from a customer Payment: update customer balance to reflect a payment Delivery: deliver orders (done as a batch transaction) Order-status: retrieve status of customer s most recent order Stock-level: monitor warehouse inventory Metrics: number of transactions processed per minute (tpmc), price/performance ($/tpmc) 6

7 TPC-C benchmark (1992) Transactions operate against a database of nine relational tables Transactions do update, insert, delete, and abort; primary and secondary key access. Response time requirement: 90% of each type of transaction must have a response time <= 5 seconds, except stock-level which is <= 20 seconds. 7

8 TPC-D benchmark (1995) Constructed by Transaction Processing Council (de facto industry standards body for online transaction processing performance, TPC-D is used to measure complex decision support system workload TPC-D models ad-hoc queries and concurrent updates in a multiuser environment CSCI317 sample database is the same as the database of TPC-D benchmark 8

9 TPC-D benchmark (1995) TPC-D database is generated by C language program dbgen TPC-D provide 17 query templates to be expanded by C language program qgen; the program replaces substitution variable with constants A sample query template: SELECT SUM(L_EXTENDEDPRICE*L_DISCOUNT) AS REVENUE FROM LINEITEM WHERE L_SHIPDATE >= DATE '[DATE]' AND L_SHIPDATE < DATE '[DATE]' + INTERVAL '1' YEAR AND L_DISCOUNTBETWEEN [DISCOUNT] AND [DISCOUNT] AND L_QUANTITY < [QUANTITY] 9

10 TPC-D benchmark (1995) TPC-D execution rules: (1) Power test Queries and updates are submitted in a single stream of query sets (no concurrency); each query set consist of 17 queries and it is followed by a sequence of updates (2) Throughput test Queries submitted in a number of concurrent streams versus one update stream (3) Load test Load test measures the time to go from an empty database to reproducible query runs; starts from empty database, loading the results of dbgen, building indexes, gathering statistics, and running queries 10

11 TPC-D benchmark (1995) Performance metrics Power metric ("queries per gigabyte hour") Size = 19 i = 17 Π i = * SF j = 2 QI ( i, 0 )* UI ( j, 0 ) Π j = 1 where QI(i,0) = Timing In terval for Query i, stream 0 UI(j,0) = Timing In terval for Update j, stream 0 SF = Scale Fac tor 11

12 TPC-D benchmark (1995) Performance metrics Throughput metric ("queries per gigabyte hour") Size = S * 17 T S ( ) 3600 * where: S = number of query str eams T = elapsed t ime of tes t (in seco nds) S SF 12

13 TPC-D benchmark (1995) Performance metrics Composite query per hour rating QphD Size QppD Size @ = * 13

14 TPC-H and TPC-R benchmarks (1999) TPC-H and TPC-R replaced TPC-D benchmark TPC-R is used to measure reporting load TPC-H is used to measure ad hoc-querying load TPC-H and TPC-R operate on the same database as TPC-D TPC-H and TPC-R extend the number of queries, updates and streams 14

15 TPC-W benchmark (2000) TPC-W is designed to measure performance of both hardware and software in e-commerce environment TPC-W is based on a business model that employs a shopping scenario typical of an online bookstore Metrics reported include "Web interactions per second" Configuration of TPC-W includes both database server and Web server 15