Run Your Own Oracle Database Benchmarks with Hammerora

Transcription

1 Run Your Own Oracle Database Benchmarks with Hammerora Steve Shaw Database Technology Manager Software and Services Group Date: 19-NOV-09 Time: Location: Seoul

2 Steve Shaw Introduction Database Technology Manager for Intel EMEA : Over 9 years with Intel Work with Customers and ISVs to implement optimal database solutions on Intel Architecture Previously with Oracle : Over 13 years of Oracle on Intel from Sequent on the Pentium Pro 180MHz onwards Co-author of Pro Oracle Database 10g RAC on Linux 2

3 Agenda Database benchmarking, what & why? Why run your own benchmarks? What benchmark? Designing Bespoke Tests Testing Application Servers Summary and Future Plans 3

4 Database Benchmarking, What & Why? 4

5 What is a benchmark? A benchmark is: a test or set of tests used to measure the performance of computer hardware or software* In our context : Software = Oracle database and operating system Hardware = Server platform including associated chipsets and technologies and storage *Encarta Dictionary (English U.K.) 5

6 Performance Fundamentals Moores Law: Innovations in technology allows a doubling of the number of transistors on a microchip every two years The implementation of high-k metal materials marks the biggest change in transistor technology since the late 1960s. Gordon Moore Silicon Advances Deliver: More transistors for new features and capabilities Higher performance Lower power consumption (switching and leakage) Smaller footprint Intel Today 45nm Switching power 30% relative to Intel 45nm process Transistor performance 20% and in the Future 32nm in 2009 Transistor performance Switching power 6

7 Oracle Transactions Balanced System Performance CPU advancement drives performance But : hierarchy of data is essential CPU cache Memory Interconnect Disk Based Storage Oracle OLTP I/O Impact on Performance Amdahl s Rule of Thumb 1 byte of memory and 1 byte per second of I/O are required for each instruction per second supported by a computer Threads

8 Oracle Benchmarks and the TPC Oracle recognises performance results that are audited, certified, and published by application vendors or independent performance evaluators at: ORACLE BENCHMARK RESULTS Online Transaction Processing (TPC-C) Data Warehousing (TPC-H) SAP (SAP SD) TPC = Transaction Processing Performance Council The TPC defines transaction processing and database benchmarks and delivers trusted results to the industry the computer industry's first standards body to bring order to the world of transaction processing performance measurement 8

9 Navigating Show All, Clustered or Non Clustered Results Or go to Advanced Sorting 9

10 Advanced Sorting Click Show Results Filter on DB Company and Server CPU Example 10

11 tpmc Official Benchmark Comparison Oracle TPC-C Performance on Xeon 1999 to , , ,000 78X Oracle performance per CPU 152X Oracle performance per CPU 97, 983 tpmc Oracle 11g 232,002 tpmc Oracle 11g 50,000 48,793 tpmc Oracle MHz Pentium II Xeon Processors GHz Intel Xeon QC GHz Intel Xeon QC E

12 Why Run your own benchmarks? 12

13 Benchmark Advantages and Challenges Published benchmark advantages A level playing field Comparable results Exceptional tests designed to scale Always the best starting point Published Benchmark Challenges No results for my hardware No results for my software System is much larger scale than mine Benchmark uses specific configuration and/or parameters e.g. in recent RAC benchmark GC_FILES_TO_LOCKS is set Note: Setting this parameter to any value other than the default will disable Cache Fusion processing in a Real Application Clusters environment. 13

14 Common Benchmarking Errors Undisciplined Testing Failing to have an aim for testing Changing multiple variable at a time Failing to accurately record configurations Failing to record performance statistics Making assumptions on guaranteed performance improvements Who watches the watchmen? How valid is the tool? Are tests accurate and repeatable? GIGO (Garbage In, Garbage Out) How valid are the tests, do they scale and produce consistent results? Invalid Conclusions Leaping to the wrong conclusion from test results 14

15 Oracle Transactions Benchmark Error Example 3 configurations, 1 user, higher number of transactions is better What would we recommend, probably 1, 3, 2 OLTP Performance System 1 System 2 System 3 1 (2r 5t) User Threads (ramp and test time) 15

16 Oracle Transactions Measured Results Actual Measured Results 2, 1, 3 Performance trends are not necessarily predictable OLTP Performance 200W System 1 System 2 System 3 28 (2r 5t) 24 (2r 5t) 20 (2r 5t) 16 (2r 5t) 12 (2r 5t) 8 (2r 5t) 4 (2r 5t) 2 (2r 5t) 1 (2r 5t) User Threads (ramp and test time) 16

17 Test Tool Requirements Essential Requirements Accuracy Consistency & Repeatability Scripted & Extensible Limitations of Commercial Tools Cost & Time Limitations Complexity General Purpose, non Oracle/Database Why not write a tool without the limitations? 17

18 Design Approach Design Brief Free and Open Source Easy, Scripted interface Database Servers Modular and Extensible Multi-Threaded Oracle Performance/OCI Graphical interface Portable to all Oracle Platforms Language Challenges Java, threaded and graphical but no scripted interface Perl, insufficient threading support C, threaded and performance but no scripted interface 18

19 Free and Open Source Architecture License Type OPEN Software Hammeora*:TCL/TK: Scripted Load Tests License Cost FREE OPEN TCL*: C Performance, C Extensibility FREE OPEN TK*: C Rapid GUI Development FREE OPEN TCL Threads: C An Interpreter per Thread FREE OPEN Oratcl: C Thread Safe OCI Connectivity* FREE CLOSED Oracle Instant Client/Express* FREE 19

20 Hammerora Search on Google for Oracle Load Test and hit I m Feeling Lucky Over 26,500 downloads to date Multithreaded Database and Web Testing Trace File Replay Includes Database tests based on TPC-C and TPC-H specifications Free and Open Source method to Compare and contrast Oracle Database performance on different hardware platforms Validate the impact of changes to software environments. 20

21 Installation Pre-compiled for Linux x86-64, Linux x86, Windows x86, Wizard Install All platform support by compiling TCL, Hammerora is the same on all platforms XML Configuration to tailor environment 21

22 Hammerora Overview Write or modify a test script View the output from virtual users Observe transactions and queries Interactive Console for host commands and error reports Define and observe independent virtual users Mode Options Control multiple instances from 1 Master 22

23 Scripting in TCL and Oratcl See the oratcl website for using oratcl Simple example: set connect set lda [oralogon $connect] set curn1 [oraopen $lda ] set sql1 "select user from dual" orasql $curn1 $sql1 Oratcl can be used for bind variables, ref cursors, LOBs etc and is thread-safe 23

24 What Benchmark? 24

25 What Benchmark? The TPC publish their specifications at no cost to the public? You can learn about the recognised official benchmarks and run your own at the same time Hammerora includes built-in benchmarks based on the TPC-C (OLTP) and TPC-H (DSS) specifications to be run in any Oracle database environment Note : Hammerora does NOT in any way implement a real TPC-C or TPC-H benchmark Do NOT compare the results of your tests with official results they are not measured the same! 25

26 What is TPC-C? TPC-C measures the performance of database servers in an On Line Transaction Processing (OLTP) environment Simulates terminal operators executing transactions against a database Emulates an order-entry environment Represents a wholesale supplier with distributed warehouses Benchmark would scale as company expands by adding new warehouses Performance metrics Performance: Transactions per min (tpmc) Price-Performance: $/tpmc Strict reporting rules - TPC-C Company Structure COMPANY WAREHOUSE -1 WAREHOUSE - DISTRICT -1 DISTRICT K CUSTOMER CUSTOMER Company A Wholesale supplier - CUSTOMER Operates from a set of warehouses A warehouse supports a set of sales districts Each district supports a set of customers Customers (clients over network) execute orderentry transactions 26

27 TPC-C Schema Warehouse W W_ID 100K Stock W*100K S_W_ID, S_I_ID W Item 100K (fixed) I_ID 10 District W*10 D_W_ID, D_ID 3+ Order-Line W*300K+ OL_W_ID, OL_D_ID, OL_O_ID, OL_NUMBER 3k Customer W*30K C_W_ID, C_D_ID, C_ID Order W*30K+ O_W_ID, O_D_ID, O_ID 0-1 New-Order W*9K+ NO_W_ID, NO_D_ID, NO_O_ID History W*30K+ None, no need to uniquely identify a row Table Name cardinality Primary Key one-to-many relationship 27

28 Hammerora TPC-C for OLTP Workload The TPC-C Specification includes full definition for User Interface, menu response time, transaction response time, keying and thinking time Hammerora implements the database side only of the TPC-C specification What? : to produce consistent and repeatable transaction results from multiple runs of the application on the same hardware and software configurations Why? : determine relative performance values Also TPC-C for MySQL 28

29 Transactions 5 Transactions implemented as PL/SQL packages: New-order: receive a new order from a customer: 45% Payment: update the customers balance to record a payment: 43% Delivery: deliver orders asynchronously: 4% Order-status: retrieve the status of customer s most recent order: 4% Stock-level: return the status of the warehouse s inventory: 4% Transactions do OLTP mix of select, update, insert and delete Hammerora TPC-C Stored procedures are written as closely as possible to the examples in the TPC-C specification The Stored procedures are not tuned for performance 29

30 TPC-C Schema Creation Select benchmark from menu Select Schema options Define Schema options Create the Schema Load Completing Creating Confirm 30

31 TPC-C Workload TPC-C Driver Script Virtual Users Run the test Create Virtual Users 31

32 Measuring Transactions per Minute Measures Oracle transactions NOT tpmc select sum(value) from v$sysstat where name = 'user commits' or name = 'user rollbacks ; 32

33 AWR Workload Takes AWR reports and reports transactional throughput Timed Tests, first user takes snapshots 33

34 Oracle Transactions Performance Profile Test System Config 2 Typical Oracle OLTP Performance Profile Test System Config 1 Baseline Previous Generation Threads

35 AWR Report Highlights (or use Grid Control) WORKLOAD REPOSITORY report for DB Name DB Id Instance Inst num Startup Time Release RAC NE NE 1 10-Mar-09 06: NO Snap Id Snap Time Sessions Cursors/Session Begin Snap: Mar-09 02:05: End Snap: Mar-09 02:10: Elapsed: DB Time: 5.13 (mins) (mins) 35

36 Load Profile AWR Report : Load Profile Per Second Per Transaction Per Exec Per Call DB Time(s): DB CPU(s): Redo size: 42,347, ,124.1 Logical reads: 851, Block changes: 261, Physical reads: Physical writes: 8, Transactions per second User calls: 12, Parses: 6, Hard parses: W/A MB processed: 759, Logons: Executes: 170, Rollbacks: Transactions: 8,

37 Top 5 Timed Events (AWR) WORKLOAD REPOSITORY report for A good first indication of CPU usage Event Waits Time(s) Avg wait (ms) % DB time Wait Class DB CPU 1, log file sync 687, Commit db file sequential read 41, User I/O db file scattered read latch: cache buffers lru chain 2, User I/O Other 37

38 Redo Throughput Per Second Transactions: 4,023.7 Event Waits Time(s) Avg wait (ms) % DB time Wait Class log file sync 894,491 5, Commit DB CPU 2, db file sequential read enq: TX - row lock contention latch: In memory undo latch 82, User I/O 17, Application 44, Concurrency Subject: WAITEVENT: "log file sync" Reference Note Doc ID: Type: REFERENCE Modified Date : 14-JUL-2009 Status: PUBLISHED Tune LGWR to get good throughput to disk. eg: Do not put redo logs on RAID 5 38

39 Redo Throughput and CPU Per Second Transactions: 8,264.3 Event Waits Time(s) Avg wait (ms) % DB time Wait Class log file sync 1,795,089 5, Commit DB CPU 2, db file sequential read enq: TX - row lock contention log file switch completion 162,468 2, User I/O 23, Application 1, Configuration Identical Storage Subsystems Identical Memory Identical OS and Oracle Database CPU Upgrade (LGWR also requires CPU) Upgrading the CPU enabled more than 2X throughput increase for redo 39

40 AWR Report : SQL ordered by CPU Time = 33 Elapsed Time = 44 40

41 Buffer Cache Sizing Buffer Pool Advisory Only rows with estimated physical reads >0 are displayed ordered by Block Size, Buffers For Estimate Back to Advisory Statistics Back to Top P Size for Est (M) Size Factor Buffers for Estimate Est Phys Read Factor Size Factor 1 is Buffer Cache Size Estimated Physical Reads D 1, , ,597,564 D 2, , ,690,979 D 3, , ,127,712 D 5, , ,508,753 D 6, , ,160,117 D 7, , ,915,528 D 8, ,104, ,736,935 D 10, ,262, ,598,715 D 11, ,420, ,490,278 D 12, ,578, ,403,674 D 12, ,585, ,399,770 D 14, ,735, ,335,253 D 15, ,893, ,277,568 D 16, ,051, ,231,349 41

42 What is TPC-H? TPC-H measures the performance of Decision Support Systems (DSS) used in Business Analysis A DSS is a computer program application that analyzes business data and presents it so that users/managers can make business decisions more easily Eg. Supply and Demand management, Customer satisfaction study, MSS study Portrays the activity of a wholesale supplier Issues a set of ad-hoc complex business queries and measures performance Categorizes results based on database sizes 100GB, 300GB, 1TB, 3TB, 10TB Performance metrics Performance: Queries Per Hour H (QphH) Price-Performance: $/QphH DSS Decision Support System Business Analysis Business Operations Source: TPC-H Overview Decision Makers DSS Queries DSS Database 100GB, 300GB, 1TB, 3TB, 10TB OLTP Database OLTP Transaction s TPC-H TPC-C 42

43 TPC-H Schema LineItem SF*6000K L_ORDERKEY L_LINENUMBER Orders SF*1500K O_ORDERKEY Customer SF*150K C_CUSTKEY PartSupp SF*800K PS_PARTKEY PS_SUPPKEY Supplier SF*10K S_SUPPKEY Nation 25 N_NATIONKEY Part SF*200K P_PARTKEY Region 5 R_REGIONKEY Table Name cardinality Primary Key one-to-many relationship 43

44 Query Streams Number of Query Streams fixed according to scale factor SF ( Scale Factor ) S (Streams) Each stream executes a query set of Q1 to Q22 ad-hoc queries Simultaneous data refresh sets No requirement for client systems with query and refresh sets able to run directly on the database server Benchmarks performed a one scale factor cannot be compared to those at another 44

45 Definition Composite Query-per-Hour Performance Metric More complex than TPC-C with 2 distinct tests 3 aspects of the capability of the system to process queries: 1. Database size 2. Query processing power of queries in a single stream 3. Total query throughput of queries from multiple concurrent users Size Power@ Size Throughput@ Size Throughput test always follows Power test 45

46 Example TPCH Query : Query 1 of using as a seed to the RNG select l_returnflag, l_linestatus, sum(l_quantity) as sum_qty, sum(l_extendedprice) as sum_base_price, sum(l_extendedprice * (1 - l_discount)) as sum_disc_price, sum(l_extendedprice * (1 - l_discount) * (1 + l_tax)) as sum_charge, avg(l_quantity) as avg_qty, avg(l_extendedprice) as avg_price, avg(l_discount) as avg_disc, count(*) as count_order from lineitem where l_shipdate <= date ' ' interval '69' day (3) group by l_returnflag, l_linestatus order by l_returnflag, l_linestatus; 46

47 TPC-H Creation and Execution Select TPC-H Select Schema Options and build TPCH Driver Script 47

48 AWR Report : Load Profile Emphasis on Reads Transactions are Zero 48

49 AWR Report : Top 5 Timed Events Top Event Direct Path Read Emphasis on PGA Direct path reads are generally used when reading directly into PGA memory (as opposed to into the buffer cache). Used for Parallel Query 49

50 AWR Report : Tablespace IO Stats Average Read 23.91ms IO Stats shows emphasis on Read performance Average Read time a useful indicator 50

51 AWR Report : PGA Memory Advisor Size Factor 1 is current PGA Target Size 51

52 Script Output : Query Set Completion Time Hammerora Wed Nov 08 11:13:09 GMT Standard Time tid000011c8:executing query 1 tid000011c8:query 1 completed in microseconds per iteration tid000011c8:executing query 2 tid000011c8:query 2 completed in microseconds per iteration tid000011c8:executing query 3 tid000011c8:query 3 completed in microseconds per iteration tid000011c8:executing query 4 tid000011c8:query 4 completed in microseconds per iteration tid000011c8:executing query 5 tid000011c8:query 5 completed in microseconds per iteration tid000011c8:completed 1 query set(s) in to = 157 seconds 52

53 Performance Profile Throughput Test on 100GB Dataset 8 Cores : Exadata 8 Cores : Exadata 1 User of 5 8 Cores : Fibre Channel 1 User 8 Cores : Fibre Channel 1 User of Query No Comparison of Exadata showed impressive performance against existing storage configuration Note the new 11gR2 in memory parallel query set PARALLEL_DEGREE_POLICY to AUTO

54 Designing Bespoke Tests 54

55 Generate Oracle Trace Files SQL TRACE normally input to tkprof for session profiling Use SQL Trace event at Level 4 for bind variables SQL> ALTER SESSION SET EVENTS '10046 trace name context forever, level 4'; SQL> ALTER SESSION SET EVENTS '10046 trace name context off'; Trace File written to USER_DUMP_DEST with name ora_spid.trc 55

56 Replay Traced Workload Convert to Oratcl Load trace file 56

57 Testing Application Servers 57

58 Tclwebtest Recorder Installation Download and install Tclwebtest recorder into firefox browser 58

59 Record and Replay Web Session Record a browsing session Load into Hammerora Replay web session 59

60 Summary and Future Plans Fully featured today and used worldwide Future Plans: TPC-E Enhanced trace file conversion including memory dumps (date datatype already done) TPC-H build and replay tuning Enhanced Web testing Enhanced builds and packaging Migrate test workloads between Oracle and MySQL 60

61 Questions? 61

62 Disclaimers Note : Hammerora is Open Source released under the GNU public license under the framework laid out by the Intel Linux User s Group Hammerora is NOT an Intel software product Hammerora is is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. 62

63 Disclaimers Information in this document is provided in connection with Intel products. No license, express or implied, by estoppels or otherwise, to any intellectual property rights is granted by this document. Except as provided in Intel's Terms and Conditions of Sale for such products, Intel assumes no liability whatsoever, and Intel disclaims any express or implied warranty, relating to sale and/or use of Intel products including liability or warranties relating to fitness for a particular purpose, merchantability, or infringement of any patent, copyright or other intellectual property right. Intel products are not intended for use in medical, life saving, or life sustaining applications. Intel does not control or audit the design or implementation of 3rd party benchmarks or websites referenced in this document. Intel encourages all of its customers to visit the referenced websites or others where similar performance benchmarks are reported and confirm whether the referenced benchmarks are accurate and reflect performance of systems available for purchase. Intel may make changes to specifications and product descriptions at any time, without notice. Designers must not rely on the absence or characteristics of any features or instructions marked reserved or undefined. Intel reserves these for future definition and shall have no responsibility whatsoever for conflicts or incompatibilities arising from future changes to them. This document contains information on products in the design phase of development. Do not finalize a design with this information. Revised information will be published when the product is available. Verify with your local sales office that you have the latest datasheet before finalizing a design. Intel, Intel logo, Intel Core and Core Inside are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. Actual measurement results may vary depending on the specific hardware and software configuration of the computer system measured, the characteristics of those computer components not under direct measurement, variation in processor manufacturing processes, the benchmark utilized, the specific ambient conditions under which the measurement is taken, and other factors. All plans, features and dates are preliminary and subject to change without notice. * Third-party brands and names are the property of their respective owners. Copyright Intel Corporation