Separating Fact from Fiction: SAP HANA and Oracle Benchmarking Claims

Transcription

1 SAP HANA Separating Fact from Fiction: SAP HANA and Oracle Benchmarking Claims Table of Contents 3 Understanding SAP Standard Application Benchmarks and the Benchmark Process 6 Questions to Ask When Assessing Oracle Benchmark Claims 9 Appendix

2 Competing against trusted businesses empowers SAP to create innovative, technically superior products that help our customers excel. But when our competitors promote benchmark results that do not adhere to existing standards, we must set the record straight. All technology companies design cutting-edge products to address customer needs. Much engineering and design effort goes into finding new ways to make IT solutions faster and more powerful. And it s up to IT and business professionals to investigate and assess vendor claims, so they can determine which products best meet their company s needs. To help the market easily and quickly make these judgments, SAP offers standard application benchmarks. When used consistently, these benchmarks provide impartial, measurement-based ratings of standard SAP applications in different configurations with regard to operating system, database, or hardware, for example. Decision makers trust these benchmarks to provide unbiased information about product performance. When SAP standard application benchmarks are not used in a way permitted by rules and guidelines, however, it is no surprise that the results are skewed. Deviation from standard rules not only can easily misrepresent product performance, it also can confuse decision makers. To give you the knowledge needed to rise above such manufactured confusion, this document explains the benchmarking process and provides several questions you should ask about benchmark claims. For additional information about benchmark terms, see the Appendix. 2 / 10

3 Understanding SAP Standard Application Benchmarks and the Benchmark Process SAP standard application benchmarks are developed by SAP and its technology partners. These benchmarks help SAP and its partners prove the performance and scalability of SAP solutions and the hardware they run on. In addition to providing quality assurance, benchmarks are used to verify the scalability, power efficiency, and multiuser behavior of system software components, relational database management systems (RDBMSs), and business applications. SAP standard application benchmarks are monitored and maintained by the SAP Benchmark Council, which was established by SAP and its partners in This council consists of representatives from SAP and technology partners such as hardware and database vendors. It helps define the content of the benchmarks and establishes the rules for the testing procedures used by hardware and database vendors during a benchmarking run. During a run, the benchmark test places a substantial load on the hardware, software, and database system to measure performance, including CPU utilization, memory consumption, network load, functional errors, and system availability. Vendors then submit the results of their run to SAP for certification. To obtain a certification, the benchmark must follow specific rules and guidelines, such as maximum response times, error-free runs, and allowed tunings. Once the benchmark is certified by SAP on behalf of the council, it is published at Customers can use the benchmark to properly size their systems for SAP applications. 3 / 10

4 The figure below shows the steps taken by the partner (vendor) and by SAP in the benchmark process. The BW-EML Benchmark To evaluate the SAP Business Warehouse application on different configurations, we use the BW-EML benchmark, which simulates a mix of multiuser reporting workloads. This benchmark was developed considering the databases commonly used by our customers. The benchmark: Simulates loading of delta requests every five minutes while processing user queries Provides ad hoc reporting using randomized navigation paths, with the ability to change displayed characteristics Reduces total cost of ownership, using DataStore Objects and InfoCubes that conserve space for reporting activities For more information about DataStore Objects and InfoCubes, see the Appendix. Figure: Benchmark Certification Process Prepare benchmark run Run benchmark Submit benchmark Analyze benchmark data Approve results Certify benchmark Verify platform dependencies restrictions with SAP Set up hardware Install software (OS, DB, and SAP software components) Obtain benchmark environment Install benchmark tools Collect statistics records and log files Verify integrity of files Fill out benchmark report sheet completely Get approval from own marketing for correct server and environment description Request an SAP Mobile Documents solution share at benchmark@sap.com Submit data to SAP using SAP Mobile Documents and notify SAP about completed upload Completed statistics records and all log files Send notice of receipt Notice of receipt Check data for completeness and plausibility Perform cross-checks and dual control Ask for additional details, if required Request approval from SAP, hardware partner, DB vendor, CPU vendor, and others Approval request mail Send reply: approval or rejection Approval or rejection mail Create certification and archive relevant benchmark data Send certificate on behalf of SAP Benchmark Council Certification mail and certificate Release publications (optional) Request benchmark data (optional) Decision to submit benchmark Benchmark report sheet Decision if benchmark is technically certifiable Decision to certify benchmark Benchmark submission kit, report sheet, and certificate Publish certification at /benchmark Partner step Deliverables, such as files, documents, and SAP step (Performance, architecture, and scalability team and SAP development specialists, if needed) Present certificate at next SAP Benchmark Council meeting 4 / 10

5 Violations of Publication Rules for the Benchmark Occasionally a vendor violates the publication rules of the BW-EML benchmark. For example, in a recent marketing campaign, Oracle made public statements claiming that its BW-EML benchmark results demonstrated that Oracle 12c Database In-Memory has twice the performance of the SAP HANA database. Because the Oracle benchmark could not be certified, the council partners unanimously decided that Oracle violated the publication rules for SAP standard application benchmarks. To learn more about best practices for the publication of benchmark information, read the publication policy for SAP standard application benchmarks. 5 / 10

6 Questions to Ask When Assessing Oracle Benchmark Claims Oracle claims that Oracle12c Database In-Memory is fast. But Oracle performed its benchmark in a nonstandard way that does not honor the requirements of the SAP standard application benchmarks. When reviewing the recent Oracle marketing campaign, keep the following questions in mind to better evaluate and understand these claims. Do I want to partner with a vendor who creates benchmarks that rely on unsupported configurations? A benchmark is a standard point of reference against which performance can be assessed. To help reviewers reach accurate conclusions, benchmarks must obey fundamental rules and recognize important technical considerations. SAP standard application benchmarks allow only those tuning measures that also can be used in production environments. However, Oracle submitted a benchmark based on a custom setup, with manual triggers and materialized views, which production environments cannot support. As a result, its benchmark cannot be certified. Can you really trust a noncertified benchmark? Am I creating risk for my business? SAP does not support the use of manually created triggers as used in the Oracle benchmark with Oracle 12c Database In-Memory software, because they can lead to inconsistencies that cannot be efficiently detected. In fact, the Global Service & Support organization at SAP does not support production systems that use manual triggers. Oracle decided to use manual triggers to improve the performance of its solution. Great benchmark performance is valuable only if it can be replicated in your particular production environment. Do you really want to improve performance speed at the risk of data consistency? Can I trust claims of 2x performance when the number of CPUs is a mystery? Oracle claims to process twice the number of navigation steps with Oracle 12c running on the Oracle Exadata Database Machine. The company says it used the same number of database CPU cores (36) that are used in the benchmarks for SAP HANA running on Dell or Lenovo servers. What Oracle doesn t mention is that the Exadata server actually has twice the number of CPU cores available for processing as were reported. The Exadata Database Machine includes the first set of CPUs in the Exadata database server and a second set of CPUs in the Exadata storage servers, which can be used to offload SQL processing from the database servers. A benchmark that does not detail the number of available CPU cores is highly untrustworthy. Benchmarks should count all available CPU cores not just the database server nodes. Claiming a 2x increase in performance while counting only half of the CPU cores capable of SQL processing is dishonest. 6 / 10

7 What is the true cost of comparable speed? Oracle s claims are based on nonequivalent systems. From a hardware perspective, the SAP HANA platform runs on well-priced hardware from 12 partners, including Dell and Lenovo. The Oracle product runs on expensive proprietary servers that make customers dependent on Oracle. On the software side, Oracle compares the speed of a one-node setup on a current Exadata server against a one-node Dell server benchmarked in October A fair comparison would be one in which the Dell server ran the same newer SAP software that was used on the Oracle Exadata machine. We do not want to speculate on the potential performance improvement if the Dell benchmark ran on the same version of the SAP NetWeaver technology platform as the Oracle benchmark. Yet the Exadata machine costs 20 to 30 times more than the Dell server and yields just 2 times the performance. Does it make sense to pay that kind of premium for little improvement in database performance? Can I live with the limitations imposed by the Oracle solution? Let s say you could achieve comparable results with the Oracle setup on Exadata. Are the limitations created by Oracle s approach worth considering? In certified benchmarks for Dell and Lenovo, SAP HANA partitioned the data across all nodes in a multinode environment. If data volume grows, you can easily scale up by adding more nodes. In comparison, Oracle replicates all data into each available node. In the eight-node configuration, this process leads to an eightfold duplication of data. Don t forget: your total database size is limited by the memory in a single node. It s also worth noting that Oracle has published results for only 1 billion initial records. SAP HANA benchmarks are available for data loads of up to 10 billion initial records, indicating its ability to scale as your business grows. Is potentially comparable speed worth a solution that cannot scale to meet your changing needs? Can I afford the cost of complexity? SAP HANA processes both transactions and analytics in memory with a single data copy. No duplicate data copies and no manual effort are required. In contrast, Oracle Database In-Memory stores transactional data on disk. To analyze the data faster, database administrators must manually duplicate tables from disk into memory. Then administrators must selectively drop analytical indexes, so they do not impact transactional application performance. This architecture increases administrative effort, data duplication, and hardware investment. Because it requires more manual effort and additional hardware, the switch to Oracle s in-memory option is costly and complex. In contrast, our simpler in-memory data approach streamlines data administration, improves performance, increases accuracy, and reduces your hardware investment. Simpler is better. Can your business afford to miss out on the benefits of a modern, in-memory solution? 7 / 10

8 How much confidence do I have in a solution with an unknown number of reference customers? Sometimes it s great to be first the first person to climb Mt. Everest or the first person to walk on the moon. But when it comes to your business, do you want to risk it all just to be first? Put your confidence where it belongs. Solution SAP Business Warehouse powered by SAP HANA SAP Business Suite powered by SAP HANA Number of Customers * 2,400 1,800 SAP S/4HANA suite 900+ Other solutions 2,100 Total: SAP HANA customers 7,200+ Total: Oracle 12c Database In-Memory customers running SAP applications? Discover Why SAP HANA Is the Better Choice SAP HANA enables transactional and analytical processing against a single copy of data, captured from a variety of data sources. It processes data in memory, enabling rapid analysis on fresh data without requiring materialized views. And because it aggregates data on the fly without the need for physical tables to store summarized data, SAP HANA increases business agility with real-time insight while reducing hardware complexity and IT costs. Instead of focusing on pure SQL acceleration, SAP HANA supports business innovation. Find Out More To learn more about certified benchmarks and SAP HANA performance and how your business can benefit visit us at 8 / 10

9 Appendix The following terms are used by the BW-EML benchmark. Data Model To ensure that the database can efficiently use both InfoCubes and DataStore Objects (DSOs) for reporting, the data model for the BW-EML benchmark consists of three InfoCubes and seven DSOs, each of which contain the data produced in one specific year. The three InfoCubes contain the same data (from the last three years) as the corresponding DSOs. Both object types include the same set of fields. The InfoCubes include a full set of 16 dimensions, which comprise 63 characteristics, with cardinalities of up to 1 million values and one complex hierarchy. To simulate typical customer data models, the InfoCube is made up of 30 key figures, including those that require exception aggregation. In the data model of the DSOs, the high-cardinality characteristics are defined as key members, while other characteristics are modeled as part of the data members. Data Volumes To test hardware configurations of various sizes, the BW-EML benchmark can be executed with different data volumes. The smallest configuration defined in the benchmark rules uses an initial load of 500 million records. (This load includes 50 million records for each InfoCube [three] and DSO [seven].) The records are loaded from ASCII flat files with a total record length of 873 bytes each. Larger-volume configurations for this benchmark include initial load volumes of 1 billion, 2 billion, or more in increments of 1 billion records. In each of these configurations, the benchmark run must load a total number of records that equals one-thousandth of the initial number of records. This volume is in addition to the initial data load. The high-load phase of the benchmark must run for at least one hour. During this time, the delta data must be loaded in intervals of five minutes. The same number of records must be loaded in each InfoCube and DSO. Query Model The BW-EML benchmark defines reports for each of two MultiProviders. A MultiProvider is a type of InfoProvider, a business intelligence object into which data is loaded. MultiProviders combine data from a number of InfoProviders and make it available for analysis purposes. The MultiProvider itself does not contain any data. Its data comes entirely from the InfoProviders on which it is based. MultiProviders include one report for the three InfoCubes and another report for the seven DSOs. The respective reports on both MultiProviders are identical, which leads to the creation of two sets of the following four reports: Report Q001: Customer-based reporting Report Q002: Material-based reporting Report Q003: Sales area based reporting Report Q004: Price comparison reporting 9 / 10

10 The reports select data for one particular year, randomly picking the InfoCube or DSO that contains the data. Further navigation steps are performed within each report. Each of these steps results in an individual query and database access point. Although the first three reports share similar navigation patterns, the filter and drill-down operations are randomized to address the demand for ad hoc types of queries. To make sure that the benchmark accesses different partitions of data, the benchmark uses random values for filter parameters. By randomly choosing characteristics for drill-down or other slice-and-dice operations, the benchmark ensures that a huge number of different characteristic combinations are covered in a multiuser reporting scenario. To guarantee a high degree of reproducibility of reporting results, the benchmark groups characteristics by their respective cardinalities. Only characteristics of the same cardinality are considered for a randomized operation. Multiuser Workload A script controls the multiuser reporting workload in the BW-EML benchmark. The script can be used to define the number of simulated users. Each simulated user logs on to the system and then executes all eight reports and ad hoc navigation steps consecutively, resulting in a total of 40 ad hoc navigation steps. After finishing all the predefined steps, the user logs off and then starts the next loop with a new log on. Simulated users are ramped up at a pace of one user logon per second. Once all configured users are running, the benchmark control environment automatically initiates a process chain that controls the delta load, which is scheduled every five minutes. After a high-load phase of at least one hour, the simulated users are ramped down and the delta loads finish. A control program checks whether the correct number of records has been uploaded during the benchmark run and whether uploaded records are visible in the report results. The most important figure reported in this benchmark run is the number of ad hoc navigation steps that are successfully executed per hour. Openness Because the BW-EML benchmark is designed for openness, it can be performed on all databases that we support to run SAP software. In fact, other vendors have published results for databases other than the SAP HANA database, proving that BW-EML is an open benchmark. The benchmark does not require vendors to tune the database to achieve maximum performance. This makes the benchmark fair for all vendors, as long as everyone adheres to the rule that no unsupported tunings are allowed. 10 / 10 Studio SAP 40939enUS (15/11)

11 No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP SE or an SAP affiliate company. SAP and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP SE (or an SAP affiliate company) in Germany and other countries. Please see for additional trademark information and notices. Some software products marketed by SAP SE and its distributors contain proprietary software components of other software vendors. National product specifications may vary. These materials are provided by SAP SE or an SAP affiliate company for informational purposes only, without representation or warranty of any kind, and SAP SE or its affiliated companies shall not be liable for errors or omissions with respect to the materials. The only warranties for SAP SE or SAP affiliate company products and services are those that are set forth in the express warranty statements accompanying such products and services, if any. Nothing herein should be construed as constituting an additional warranty. In particular, SAP SE or its affiliated companies have no obligation to pursue any course of business outlined in this document or any related presentation, or to develop or release any functionality mentioned therein. This document, or any related presentation, and SAP SE s or its affiliated companies strategy and possible future developments, products, and/or platform directions and functionality are all subject to change and may be changed by SAP SE or its affiliated companies at any time for any reason without notice. The information in this document is not a commitment, promise, or legal obligation to deliver any material, code, or functionality. All forward-looking statements are subject to various risks and uncertainties that could cause actual results to differ materially from expectations. Readers are cautioned not to place undue reliance on these forward-looking statements, which speak only as of their dates, and they should not be relied upon in making purchasing decisions.