A Composite Benchmark for Online Transaction Processing and Operational Reporting

Transcription

1 A Composite Benchmark for Online Transaction Processing and Operational Reporting Anja Bog, Jens Krüger, Jan Schaffner Hasso Plattner Institute, University of Potsdam August-Bebel-Str 88, Potsdam, Germany {anja.bog, jens.krueger, jan.schaffner Abstract Up-to-date data is of immense importance for operational reporting. Global enterprises require such a high throughput during daily operations that reporting systems had to be separated from the transactional system to avoid inhibiting performance. These architectures, however, do not provide the required reporting flexibility as the data set is a pre-defined subset of the actual data and updated only at certain time intervals, e.g. nightly. The composite benchmark for online transaction processing (OLTP) and operational reporting, henceforth CBTR, provides means to evaluate the performance of enterprise systems for a mixed workload of OLTP and operational reporting queries. Such a system offers up-to-date information and the flexibility of the entire data set for reporting. CBTR provokes the conflicts that were the reason for separating the two workloads on different systems. In this paper we introduce the concepts of CBTR, which is based on the original data set and real workloads of an existing, globally operating enterprise. 1. Introduction OLTP systems are the backbone of today s information systems supporting the daily operations[12]. All incoming requests and information of customers, suppliers and other business partners, as well as internal production processes have to be recorded, monitored and processed. Therefore, throughput is a key element. Online analytical processing (OLAP) systems support decision making processes and determining future strategies on the basis of analyzing data collected during daily operations. Inmon [8] distinguishes between two kinds of reporting: operational and informational. Operational reporting includes the most recent information within an OLTP system of an enterprise supporting the day-to-day activities in detail. Informational reporting supports strategic and long-term decision-making, whereas huge amounts of data is summarized. Operational reporting and OLTP are closely related in terms of what data set is used for processing. Since, both workloads interfere with each other it is common to provide a dedicated system for each. Specialized data stores, called operational data store (ODS)[7], support operational reporting and data warehouses [6] support informational reporting without impeding the performance of the OLTP system. Advances in hardware technology, e.g. the vast increase of CPU speed and memory sizes, as well as new trends in data storage management, e.g. column-oriented data store architectures, also called column stores, attract the idea that combining the OLTP workload with the operational reporting workload within the same system might become possible. As a contribution of this paper, we present a benchmark to measure the impact of a combined workload consisting of OLTP transactions and operational reporting queries. For this we introduce a scenario with an integrated data schema relevant for both. CBTR does not use generated data. It is entirely based on the data schema, transactions and operational reports taking place during the daily operations of a real enterprise. The benchmark will run with the actual data of the enterprise, though a data generator will be provided for the public version. CBTR also introduces two novel scales besides varying the data set size, i.e. altering the data schema and the mixture of the workload. The remainder of this paper is structured as follows: In Section 2 we introduce the methodology for creating a new benchmark out of the systems of an existing enterprise. Section 3 introduces the scenario and data schema of CBTR, as well as the transactions for OLTP and queries for operational reporting on top of this data schema. Furthermore, the problems arising from mixing the different workloads are discussed and we show how they are incorporated in CBTR. In Section 4 we examine the effects of using real company data of daily business operations for the benchmark instead of generated data and introduce the different

2 scales along which CBTR can be modified. Furthermore, existing database performance benchmarks for transaction processing and analytics are reviewed and their applicability to our use case of a combined workload is analyzed. Section 5 concludes the paper. 2. Methodology Instead of combining existing standardized benchmarks and adjusting these to obtain a mixed workload, we are creating an entirely new benchmark closely resembling a realworld scenario filled with real enterprise data. The first step toward a realistic benchmark was finding an enterprise sufficiently large to ensure the scaling of database size, e.g. starting from a small database consisting only of several gigabytes of data of one division scaling up to a data set worth several terabytes of all global divisions together. Second, from the variety of scenarios within the enterprise, e.g. order-to-cash management, procure-to-pay management, or supply chain planning, one is chosen for the benchmark. We decided to use the order-to-cash cycle as the basis for the benchmark. This scenario is equally important for daily operations, e.g. incoming orders, outgoing invoices, and finally incoming payments, as it is for operational reporting, e.g. analyzing the levels of order fulfillment, determining all open orders of customers to ensure timely deliveries, or finding late payments to trigger dunning. The next step is the extraction of the data schema for the benchmark from the enterprises OLTP system. In the beginning the schema and data is kept without changes, but cut cleanly out of the entire environment to achieve a closed setting. How the schema, and consequently the data, will evolve with the progressing of the benchmark is discussed in Section 4.2. Following the OLTP transactions on top of the orderto-cash data schema are extracted including the intertransaction dependencies, for example, an invoice can usually only be sent to the customer if all ordered items have been delivered. Therefore, the transactions recording the delivery of all the line items that will be invoiced should have happened before the transaction creating the invoice in the system. In the same way, the operational reporting queries relevant for the order-to-cash cycle will be extracted from the ODS, which is a part of the OLAP system of the enterprise. Since operational reporting needs data on a very detailed level, most suitably line item level, we found the queries using very similar structures as in the OLTP system. Using the data warehouse for such operational reports requiring line item data is not a preferred use case as data warehouses do not usually hold data on item level. Requiring data on line item granularity leads to an explosion in data set size as aggregates inherent to the data warehouse architecture have to be built for lots of combinations. 3. Designing the Benchmark OLTP and operational reporting are two fundamentally different workloads interfering with each other. In this section we propose the logical database schema, the OLTP transactions and operational reporting queries for the benchmark. We afterwards analyze the conflicts that are met when mixing transaction processing and operational reporting workloads. Then we show how these conflicts are built into the benchmark to achieve a full picture of the performance of an architecture processing both workloads in parallel Database Schema The database schema provided in this section will be the first notch on the scale of varying the underlying logical database schema from being optimized for OLTP to being optimized for OLAP as discussed in Section 4.2. The process underneath the order-to-cash scenario consists of several steps. These are the creation of a sales order, the delivery of ordered items, billing delivered items and clearing incoming payments. Therefore, the database schema consists of table sets belonging to one of these steps. Figure 1 depicts a very simplified overview of the schema showing only an excerpt of the most important tables and attributes. Figure 1. Order-to-cash database schema

3 Regarding the general structure, each set of tables contains a header and line item table, e.g. sales document header and sales document line item. The header table contains general information applicable to the entire set of line items belonging to it. Information stored in the sales document header is for example the order date, requested delivery date, and sold-to-party. A line item belongs to exactly one header and a header contains at least one line item. Specific data contained in the sales document line item table is the ordered material, quantity, allowed deviation from the ordered quantity, weight and volume information. Sales document partner may contain additional information about parties with specific roles for the line items. These roles are, e.g. sold-to-party, ship-to-party, payer, or vendor of a material. Consequently, if the ship-to-party is given on line item level, differing from the information given in the header, the header will be overridden for this case. The sales document business data table holds data for each line item only if a deviation from the data given in the header occurs. The other table sets, delivery tables, billing tables and financial tables show a similar behavior, which will not be discussed in the same detail. The different table sets are connected via their line items. The path of a line item can be traced from the order, over the delivery to the final payment. As a result, no one-to-one relationship exists between a sales order document and a delivery document. In fact, a delivery document may either span several sales order documents or may be only a partial delivery, meaning that the line items of one sales order document are split over several deliveries. This is also true for billing documents, where a customer may trigger several orders, but wants to pay after all have been delivered. The alternative, where a billing document covers only part of a delivery is rare, but also possible. Master data tables have been omitted in the overview of the data schema. These are customer detail and material detail tables. Further tables omitted in the overview are status tables. These hold information about the progress of the entire process The Transactions and Queries The transactional side of CBTR consists of the following transactions with write access: Creation of new sales order documents including line items, entering the customer details, ordered materials and quantities and negotiating a delivery date: Here checks have to be run if all selected materials, and the customer is maintained in the system. Creation of delivery documents based on sales orders: This transaction checks if all items to be delivered have been ordered beforehand and are sent to the same receiving party. Creation of billing documents: check if all referenced line items have been delivered before issuing the bill. Creating billing documents triggers the creation of according documents in the accounting system, where the items are marked as open. Open line items are those items that have been delivered, but not yet paid. Recording of payments and clearing open line items: This transaction includes pulling all open line items from the database and finding those that are referenced in an incoming payment document to clear them. Read-only OLTP transactions are also included, e.g. displaying a complete sales order document, delivery document, billing document, or accounting document, selecting by key, showing all not yet billed, but delivered items, and displaying customer and material details. The operational reporting queries include the following: Two queries for ensuring customer satisfaction, that determine the entire order processing time starting from the incoming order till the last item has been delivered to the customer, and the percentage of sales orders having been shipped completely and on-time. Determine the average time between the delivery of items until the payment is recorded, also known as days sales outstanding. The daily flash query identifies today s sales revenues, grouped, e.g. by customer, product, or region. Determine the liquidity based on payment due dates and clearing time of past payments. This can be done per customer or over all customers. To achieve a realistic mixture of the workload, the detailed interactions in the systems of the enterprise have to be analyzed and ported to CBTR. This includes statistics for each transaction and query, such as how many orders are created within a certain division in average per minute, and how many users are working on the system in parallel. Most systems provide such capabilities that track user interactions, database transactions, and system loads, which can be exploited here. This, however, is still future work Mixing the Workload French [5] gives a characterization of the different workloads of OLTP and reporting. OLTP transactions are characterized by simple, mixed read and write operations. These are usually row-based, retrieving a large number of columns. For example, an entire sales order document including all its attributes and line items with details is displayed or updated. OLTP transactions are furthermore highly selective, meaning that exactly one sales order document and its line item entries are touched, instead of a huge set of them. Reporting queries on the other hand are mainly readonly, complex and the data sets that are touched by a query

4 are large. In relation to OLTP transactions, they are relatively long-running due to their low selectivity and only a small number of columns is usually retrieved. For example, a query computing the average number of line items in a sales order only needs the identifier column of the sales order table and the identifier and line item position columns of the sales order line item table. Several conflicts arise from the different behaviors of both workloads. One of them is resource contention. Due to the difference in running times, OLTP transactions might queue up while a reporting query is processed. A solution to this problem is given by emerging multi-core and multiprocessor architectures, which are henceforth widely available. If a reporting query reads all the values of a column in a table, OLTP transactions trying to update single values or add new records are blocked for writing and have to wait until the reading query has released its locks on the column. Inserts during reporting queries are also an issue that has to be taken care of. This can either be done through employing snapshot isolation or relaxing the isolation level of reporting queries to non-repeatable read, which causes no trouble as reporting queries are read-only. Both conflicts are going to occur in the benchmark inherently as we are operating on the same data set. Consequently, they have to be solved by the architecture under test. OLAP queries directly on top of traditional OLTP row store implementations have been proven to be slow, as it includes lots of join operations and scans of the entire table. Not just the column of interest are read, but all other columns, too, since row stores usually operate on the granularity of entire tuples [4]. This issue has been targeted from two sides: First, optimizations have been developed restructuring the logical data schema, i.e. introducing the star and snowflake schema. These ensure a lower read overhead for high volume reporting queries. The benchmark will tackle these optimizations in future versions with the goal of figuring out a schema for a combined workload with the least overhead for both sides. The latest trend of optimization for a specific workload is going away from the traditional relational implementations. The physical layout of data is changed and the data access engine is adapted accordingly, like has been shown in the C-Store project [11]. This, however, is transparent for the benchmark. 4. Unique Properties of CBTR The properties contributing to the uniqueness of CBTR, justifying its existence complementary to standardized benchmarks, are discussed in this chapter. The first is, that CBTR is entirely based on real data and the second is that novel scales, besides data set size are introduced Real Data CBTR is not just defined closely resembling the behavior of original systems of an existing enterprise, but actually operates with real data instead of generated data. Most data generators consider statistical distributions of data in real enterprises. These, however, still lead to idealistic data as real data usually deviates in one way or the other from statistical distributions. Therefore, generated data sets never achieve the same behavior as really operating with original data. The best way of getting the real picture is still testing within the enterprise, which however is not possible, due to high availability requirements and a costly implementation of the benchmark activity. Consequently, CBTR, taking a snapshot of real systems, is as close to a real scenario as possible. Since no data will be generated, the original data set has to be split in two parts. One is loaded into the database upfront and the other set is used to feed the OLTP transactions with real data to keep the entire data set realistic Three Scales As mentioned before, CBTR introduces three scales to focus on different aspects. The first is the commonly used one for varying the data set size. The special case here is that scaling up is not as easy as generating a new data set with more branches and customers. Rather, the original data set has to be cut along its business boundaries, e.g. by division. Starting with one division and adding more divisions going along will increase the entire size of the database. Divisions are, however, not equal in size. As a result the scaling along divisions may not be done in a linear fashion. The second scale is the variation of the logical database schema from being optimized for transaction processing to being optimized for reporting. Currently, the two ends of this scale are clear. The OLTP-optimized schema was described earlier in this paper in Section 3.1 and the end of this scale resulting in a star or snowflake schema optimized for the reporting queries is also conceivable. It can be copied out of the ODS in the enterprises data warehouse system. One constraint, which has to be ensured is that all OLTP transactions of CBTR are still feasible on this schema. Future work here includes to find characteristics to achieve meaningful alterations of one schema toward the other, and as a result adapting the data set without loosing its authenticity. The last scale is a variation of the workloads starting from pure OLTP, going over the realistic mixed workload, which has been observed, and ending in a pure reporting workload. Applying the pure OLTP workload on the OLTP schema and doing the same for the reporting workload, will result in a baseline to compare the mixed workload with to

5 measure the performance losses. A side effect of CBTR is the creation of a data schema, which is applicable for the mixed workload without much performance loss compared to the baselines and the validation of this schema Related Work For each, OLTP and OLAP, benchmarks exist to validate the performance of new architectures. The transaction processing performance council (TPC) 1 benchmarks have become the standard benchmarks for OLTP and OLAP. TPC s currently active benchmarks are TPC-C[1] and its successor TPC-E[2] for OLTP and TPC-H[3] for OLAP. A new benchmark for OLAP, TPC-DS[10], is now in the review phase. These standard benchmarks could be applied to a combined architecture for OLTP and operational reporting, simply running them in parallel, but this would only lead to a partial picture of the actual performance of such a system measuring only the effects of hardware resource contention. The reason for this is that the underlying data schemes of the different benchmarks differ to a great extent and cannot be integrated to create a sufficiently relevant and still realistic data schema for both. Data access conflicts are, however, characteristic for a combined workload. Another issue of the standard benchmarks is that they still do not reflect the actual schemes used for operations, but an ideal version. According to Othayoth and Poess[9] real schemes suffer from a considerably larger number of tables and columns. In CBTR, the table set is cut down, but the number of columns in the tables is quite large, varying between 50 to 300 columns. Furthermore, Windsor et al [13] claim that workloads in production exhibit a wider range of behavior, than is reflected by the TPC benchmarks. Therefore, they are not representative of real workloads. 5. Conclusion In this paper we introduced the conceptual design of a new benchmark combining transaction processing and operational reporting. The detailed specification will be reported separately. We have argued why such a benchmark is unique and complements the existing portfolio of standardized benchmarks. In contrast to all of them, this benchmark is entirely based on realistic data and targets architectures capable of performing mixed workloads that have been separated onto different systems in the past. A combined system will be a huge breakthrough for globally operating enterprises. We intend to use CBTR to compare such a system with traditional OLTP and OLAP systems. Future work is still needed in specifying details of the benchmark, e.g. completing the scale varying the data 1 schemes, drawing out the OLAP optimized schema, and defining the ones in-between. Furthermore, the exact workload characteristics have to be analyzed in the real systems and applied in CBTR. References [1] TPC Benchmark C, Standard Specification, Revision 5.9. Technical report, Transaction Processing Performance Council, June [2] TPC Benchmark E, Standard Specification, Version Technical report, Transaction Processing Performance Council, April [3] TPC Benchmark H (Decision Support), Standard Specification, Revision Technical report, Transaction Processing Performance Council, February [4] M. M. Astrahan, M. W. Blasgen, D. D. Chamberlin, K. P. Eswaran, J. N. Gray, P. P. Griffiths, W. F. King, R. A. Lorie, P. R. McJones, J. W. Mehl, G. R. Putzolu, I. L. Traiger, B. W. Wade, and V. Watson. System R: Relational Approach to Database Management. ACM Trans. Database Syst., 1(2):97 137, [5] C. D. French. Teaching an OLTP Database Kernel Advanced Data Warehousing Techniques. In ICDE 97: Proceedings of the Thirteenth International Conference on Data Engineering, pages , Washington, DC, USA, IEEE Computer Society. [6] W. H. Inmon. Building the Data Warehouse (2nd ed.). John Wiley & Sons, Inc., New York, NY, USA, [7] W. H. Inmon. Building the Operational Data Store. John Wiley & Sons, Inc., New York, NY, USA, [8] W. H. Inmon. Operational and Informational Reporting: Information Management: Charting the Course. DM Review Magazine, July, [9] R. Othayoth and M. Poess. The Making of TPC-DS. In VLDB 06: Proceedings of the 32nd International Conference on Very Large Databases, pages VLDB Endowment, [10] M. Poess, B. Smith, L. Kollar, and P. Larson. TPC-DS, Taking Decision Support Benchmarking to the next Level. In SIGMOD 02: Proceedings of the 2002 ACM SIGMOD International Conference on Management of Data, pages , New York, NY, USA, ACM. [11] M. Stonebraker, D. J. Abadi, A. Batkin, X. Chen, M. Cherniack, M. Ferreira, E. Lau, A. Lin, S. Madden, E. O Neil, P. O Neil, A. Rasin, N. Tran, and S. Zdonik. C-store: A Column-oriented DBMS. In VLDB 05: Proceedings of the 31st International Conference on Very large Databases, pages VLDB Endowment, [12] M. Vieira and H. Madeira. A Dependability Benchmark for OLTP Application Environments. In VLDB 03: Proceedings of the 29th International Conference on Very Large Databases, pages VLDB Endowment, [13] Windsor W. Hsu and Alan J Smith and Honesty C. Young. Analysis of the Characteristics of Production Database Workloads and Comparison with the TPC Benchmarks. Technical report, Berkeley, CA, USA, 1999.