TPC Benchmark TM H Full Disclosure Report for IBM xseries 350 using Microsoft SQL Server 2000 Enterprise Edition. Submitted for Review

Transcription

1 TPC Benchmark TM H Full Disclosure Report for IBM xseries 350 using Microsoft SQL Server 2000 Enterprise Edition Submitted for Review March 23, 2001

2 First Edition - March 2001 THE INFORMATION CONTAINED IN THIS DOCUMENT IS DISTRIBUTED ON AN AS IS BASIS WITHOUT ANY WARRANTY EITHER EXPRESSED OR IMPLIED. The use of this information or the implementation of any of these techniques is the customer s responsibility and depends on the customer s ability to evaluate and integrate them into the customer s operational environment. While each item has been reviewed by IBM for accuracy in a specific situation, there is no guarantee that the same or similar results will be obtained elsewhere. Customers attempting to adapt these techniques to their own environment do so at their own risk. In this document, any references made to an IBM licensed program are not intended to state or imply that only IBM s licensed program may be used; any functionally equivalent program may be used. This publication was produced in the United States. IBM may not offer the products, services, or features discussed in this document in other countries, and the information is subject to change without notice. Consult your local IBM representative for information on products and services available in your area. Copyright International Business Machines Corporation All rights reserved. Permission is hereby granted to reproduce this document in whole or in part, provided the copyright notice as printed above is set forth in full text on the title page of each item reproduced. U.S. Government Users - Documentation related to restricted rights: Use, duplication, or disclosure is subject to restrictions set forth in GSA ADP Schedule Contract with IBM Corp. Trademarks IBM is a registered trademark, and the e-business logo and xseries are trademarks of International Business Machines Corporation. The following terms used in this publication are trademarks of other companies as follows: TPC Benchmark, TPC-H, QppH QthH and QphH are trademarks of Transaction Processing Performance Council; Intel, Pentium and Xeon are registered trademarks of Intel Corporation; Microsoft and Windows are trademarks or registered trademarks of Microsoft Corporation. Other company, product, or service names, which may be denoted by two asterisks (**), may be trademarks or service marks of others. Notes 1 MHz only measures microprocessor internal clock speed, not application performance. Many factors affect application performance. 2 When referring to hard disk capacity, one GB equals one billion bytes. Total user-accessible capacity may be less. IBM Corporation TPC Benchmark H Full Disclosure Report March 23,

3 Total System Cost $158,653 IBM ^ xseries 350 with Microsoft SQL Server 2000 Composite Query-per-Hour Metric 1, GB TPC-H Rev Report Date: Mar. 23, 2001 Price/Performance $136 per QphH@100GB Database Size 100GB Database Manager Microsoft SQL Server 2000 Enterprise Edition Operating System Microsoft Windows 2000 Advanced Server Other Software Visual Studio Professional 6.0 Availability Date May 31, 2001 Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q12 Q13 Q14 Q15 Q16 Q17 Q18 Q19 Q20 Q21 Q22 RF1 RF Power Test Throughput Test Geometric Mean of Power Test Arithmetic Mean of Throughput Test Thousands Database Load Time: 06:59:58 RAID (Base Tables Only): N Configuration Processors Memory Disk Controllers Disk Drives Total Disk Storage Load Included Backup: Y RAID (Base Tables and Auxiliary Data Structures): N Total Data Storage / Database Size: RAID (All): N Intel Pentium III Xeon 900MHz / 2MB L2 Cache 1GB SDRAM ECC RDIMM II Mylex extremeraid 2000 Adapter ServeRAID-4H Ultra160 SCSI Adapter 9.1GB 15K Ultra160 SCSI Drive GB

4 Server Hardware Netfinity Rack Side Panel Kit Server Storage Description xseries 350 / 900MHz/2MB L2 Cache Pentium III Xeon 900MHz / 2MB L2 Cache Processor 1GB SDRAM ECC RDIMM II Mylex extremeraid 2000 Adapter* ServeRAID-4H Ultra160 SCSI Adapter Netfinity 4.2M Ultra2 SCSI Cable 270W Hot-Swap Redundant Power Supply E54 15 (13.8 Viewable) Color Monitor Netfinity EXP300 Rack Storage Expansion Enclosure Netfinity 9.1GB 15K Ultra160 SCSI Drive Server Software Microsoft SQL Server 2000 Enterprise Edition Microsoft SQL Server 2000 Client Licenses Microsoft Windows 2000 Advanced C Server Visual Studio Professional 6.0 Win IBM ^ xseries 350 Microsoft SQL Server 2000 Part Number RY 19K L3119 E NB 37L K L N2N G RU 19K0655 Source Large volume discount of 14% on IBM hardware; prices vary if purchased separately. Notes: Standard 3-Year and extended warranties upgraded to 7x24, 4-hour response time coverage. *10% or minimum 2 spares are added in place of on-site service; product has a 5-year return-to-vendor warranty. Pricing: 1 - IBM List Price; 2 - Microsoft Corp.; 3 - Mylex Audited by Brad Askins of InfoSizing, Inc. Subtotal $145,329 ($16,488) $29,812 5-Year Cost of Ownership: $158, GB: 1,169.7 $ / 100GB: $136 Prices used in TPC benchmarks reflect the actual prices a customer would pay for a one-time purchase of the stated components. Individually negotiated discounts are not permitted. Special prices based on assumptions about past or future purchases are not permitted. All discounts reflect standard pricing policies for the listed components. For complete details, see the pricing sections of the TPC benchmark specification. If you find that stated prices are not available according to these terms, please inform the TPC at pricing@tpc.org. Thank you Unit Price 8,199 6,189 1,689 1,872 2, , , ,549 7,300 2,399 1,079 TPC-H Revision Report Date: March 23, 2001 Qty Subtotal Subtotal Total Ext. Price 8,199 18,567 6,756 11,232 2,439 1, , $51,386 28,611 49,005 $77,616 5,549 7,300 2,399 1,079 $16,327 5-Yr. Maint. 10, ,308 0 $12,263 7,074 0 $7,074 10,475 Incl Above Incl Above Incl Above $10,475 IBM Corporation TPC Benchmark H Full Disclosure Report March 23,

5 IBM ^ xseries 350 with Microsoft SQL Server 2000 TPC-H Rev Report Date: March 23, 2001 Measurement Results: Database Scale Factor Total Data Storage/Database Size Start of Database Load End of Database Load Database Load Time Query Streams for Throughput Test TPC-H Power TPC-H Throughput TPC-H Composite Query-per-Hour (QphH@100GB) Total System Price over 5 Years TPC-H Price/Performance Metric ($/QphH@100GB) :35:16 20:35:14 06:59: ,169.7 $158,653 $136 Measurement Interval: Measurement Interval in Throughput Test (Ts) 49,434 seconds Duration of Stream Execution: Seed Query Start Date/Time Query End Date/Time RF1 Start Date/Time RF1 End Date/Time RF2 Start Date/Time RF2 End Date/Time Duration Stream /14/01 15:38:47 02/14/01 18:20:44 02/14/01 15:37:06 02/14/01 15:38:45 02/14/01 18:20:45 02/14/01 18:22:26 02:45:19 Stream /14/01 18:22:28 02/15/01 06:41:35 02/14/01 18:22:28 02/15/01 07:48:04 02/15/01 07:48:05 02/15/01 07:50:00 12:19:08 Stream /14/01 18:22:28 02/15/01 07:12:40 02/15/01 07:50:01 02/15/01 07:51:53 02/15/01 07:51:54 02/15/01 07:53:56 12:50:12 Stream /14/01 18:22:28 02/15/01 07:46:20 02/15/01 07:53:58 02/15/01 07:55:58 02/15/01 07:56:00 02/15/01 07:58:01 13:23:52 Stream /14/01 18:22:28 02/15/01 06:47:18 02/15/01 07:58:02 02/15/01 08:00:05 02/15/01 08:00:07 02/15/01 08:02:08 12:24:49 Stream /14/01 18:22:29 02/15/01 06:21:55 02/15/01 08:02:09 02/15/01 08:04:13 02/15/01 08:04:14 02/15/01 08:06:22 11:59:27 IBM Corporation TPC Benchmark H Full Disclosure Report March 23,

6 IBM ^ xseries 350 with Microsoft SQL Server 2000 TPC-H Rev Report Date: March 23, 2001 TPC-H Timing Intervals (in seconds): Query Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q12 Stream 00 1, , Stream 01 3, , , , , ,324.5 Stream 02 2, , , , , , , , ,252.1 Stream 03 2, , , , , , , Stream 04 2, , , , , , , , ,579.1 Stream 05 3, , , , , Minimum 2, , , , Average 2, , , , , , , ,055.3 Maximum 3, , , , , , , , , ,579.1 Stream ID Q13 Q14 Q15 Q16 Q17 Q18 Q19 Q20 Q21 Q22 RF1 RF2 Stream , , Stream 01 2, , , , , , Stream 02 1, , , , , Stream , , , , Stream 04 4, , , , , Stream 05 1, , , , , Minimum Average , , , , , , , , Maximum 4, , , , , , , , IBM Corporation TPC Benchmark H Full Disclosure Report March 23,

7 Table of Contents Preface... 1 General Items Benchmark Sponsor Parameter Settings Configuration Diagrams Measured Configuration Priced Configuration... 2 Clause 1: Logical Database Design Related Items Database Table Definitions Database Organization Horizontal Partitioning Replication... 3 Clause 2: Queries and Update Functions Related Items Query Language Random Number Generation Substitution Parameters Generation Query Text and Output Data from Database Query Substitution Parameters and Seeds Used Query Isolation Level Refresh Function Implementation... 4 Clause 3: Database System Properties Related Items ACID Properties Atomicity Completed Transactions Aborted Transactions Consistency Consistency Test Isolation Read-Write Conflict with Commit Read-Write Conflict with Rollback Write-Write Conflict with Commit Write-Write Conflict with Rollback Concurrent Progress of Read and Write on Different Tables Updates Not Indefinitely Delayed by Reads on Same Table Durability Failure of a Durable Medium System Crash Memory Failure... 5 Clause 4: Scaling and Database Population Related Items Cardinality of Tables Distribution of Tables and Logs Database Partition / Replication Mapping RAID Implementation DBGEN Modifications Database Load Time Data Storage Ratio Database Load Mechanism Details and Illustration Qualification Database Configuration... 6 Clause 5: Performance Metrics and Execution Rules Related Items System Activity between Load and Performance Tests Steps in the Power Test Timing Intervals for Each Query and Refresh Function IBM Corporation TPC Benchmark H Full Disclosure Report March 23,

8 6.4 Number of Streams for the Throughput Test Start and End Date/Times for Each Query Stream Total Elapsed Time for the Measurement Interval Refresh Function Start Date/Time and Finish Date/Time Timing Intervals for Each Query and Each Refresh Function for Each Stream Performance Metrics Performance Metric and Numerical Quantities from Both Runs System Activity between Tests Clause 6: SUT and Driver Implementation Related Items Driver Implementation-Specific Layer Power Run Throughput Run Clause 7: Pricing Related Items Hardware and Software Components Five-Year Cost of System Configuration Availability Dates Country-Specific Pricing Clause 9: Audit Related Items Auditor s Report Appendix A: Tunable Parameters and System Configuration Microsoft SQL Server 2000 Version Microsoft SQL Server 2000 Installation Microsoft SQL Server 2000 Startup Parameters Microsoft SQL Server 2000 Configuration Parameters Microsoft Windows 2000 Advanced Server Configuration Parameters Microsoft Windows 2000 Advanced Server Boot.ini Parameters Microsoft Windows 2000 Advanced Server Registry Parameters System Hardware Information Disk Controller Configuration Parameters ServeRAID-4H Ultra160 SCSI Adapter Mylex extremeraid 2000 Adapter Mylex extremeraid 2000 Adapter Mylex extremeraid 2000 Adapter Mylex extremeraid 2000 Adapter Appendix B: Database, Table, Index Creation and Backup, Restore, TempDB and TempLog Scripts CreateDatabase.sql (Test Database) CreateTables.sql PinBaseTables.sql CreateIndexesStream1.sql CreateIndexesStream2.sql CreateIndexesStream3.sql CreateIndexesStream4.sql CreateClusteredIndexes.sql BackupDatabase1.sql (Test Database) BackupDatabase2.sql (Test Database) RestoreDatabase1.sql (Test Database) RestoreDatabase2.sql (Test Database) ResizeTempDB.sql ResizeTempDB2.sql ResizeTempLog.sql CreateBackupDevices1.sql (Test Database) CreateBackupDevices2.sql (Test Database) CreateDeleteTables.sql IBM Corporation TPC Benchmark H Full Disclosure Report March 23,

9 CreateInsertTables.sql MoveTempDB.sql MoveTempLog.sql CreateDatabase.sql (Qualification Database) CreateBackupDevices1.sql (Qualification Database) CreateBackupDevices2.sql (Qualification Database) RestoreDatabase1.sql (Qualification Database) RestoreDatabase2.sql (Qualification Database) BackupDatabase1.sql (Qualification Database) BackupDatabase2.sql (Qualification Database) Appendix C: Query Validation Scripts and Output Validation_Query_1.sql Validation_Query_1.out Validation_Query_2.sql Validation_Query_2.out Validation_Query_3.sql Validation_Query_3.out Validation_Query_4.sql Validation_Query_4.out Validation_Query_5.sql Validation_Query_5.out Validation_Query_6.sql Validation_Query_6.out Validation_Query_7.sql Validation_Query_7.out Validation_Query_8.sql Validation_Query_8.out Validation_Query_9.sql Validation_Query_9.out Validation_Query_10.sql Validation_Query_10.out Validation_Query_11.sql Validation_Query_11.out Validation_Query_12.sql Validation_Query_12.out Validation_Query_13.sql Validation_Query_13.out Validation_Query_14.sql Validation_Query_14.out Validation_Query_15.sql Validation_Query_15.out Validation_Query_16.sql Validation_Query_16.out Validation_Query_17.sql Validation_Query_17.out Validation_Query_18.sql Validation_Query_18.out Validation_Query_19.sql Validation_Query_19.out Validation_Query_20.sql Validation_Query_20.out Validation_Query_21.sql Validation_Query_21.out Validation_Query_22.sql Validation_Query_22.out IBM Corporation TPC Benchmark H Full Disclosure Report March 23,

10 Appendix D: Seed and Query Substitution Parameters Appendix E: Refresh Function Source Code Appendix F: StepMaster Source Code Appendix G: Implementation Specific Layer and Source Code Appendix H: Price Quotations IBM Corporation TPC Benchmark H Full Disclosure Report March 23,

11 Preface The TPC Benchmark H is a decision support benchmark. It consists of a suite of business-oriented ad hoc queries and concurrent data modifications. The queries and the data populating the database have been chosen to have broad industrywide relevance while maintaining a sufficient degree of ease of implementation. This benchmark illustrates decision support systems that: Y Examine large volumes of data; Y Execute queries with a high degree of complexity; Y Give answers to critical business questions. TPC-H evaluates the performance of various decision support systems by the execution of set of queries against a standard database under controlled conditions. The TPC-H queries: Y Give answers to real-world business questions; Y Simulate generated ad-hoc queries (e.g., via a point-and-click GUI interface); Y Are far more complex than most OLTP transactions; Y Include a rich breadth of operators and selectivity constraints; Y Generate intensive activity on the part of the database server component of the system under test; Y Are executed against a database complying with specific population and scaling requirements; Y Are implemented with constraints derived from staying closely synchronized with an on-line production database. The TPC-H operations are modeled as follows: Y The database is continuously available 24 hours a day, 7 days a week, for ad-hoc queries from multiple end users and data modifications against all tables, exeat possibly during infrequent (e.g., once a month) maintenance sessions. Y The TPC-H database tracks, possibly with some delay, the state of the OLTP database through ongoing refresh functions, which batch together a number of modifications impacting some part of the decision support database. Y Due to the worldwide nature of the business data stored in the TPC-H database, the queries and the refresh functions may be executed against the database at any time, especially in relation to each other. In addition, this mix of queries and refresh functions is subject to specific ACIDity requirements, since queries and refresh functions may execute concurrently. Y To achieve the optimal compromise between performance and operational requirements, the database administrator can set, once and for all, the locking levels and the concurrent scheduling rules for queries and refresh functions. The minimum database required to run the benchmark holds business data from 10,000 suppliers. It contains almost 10 million rows representing a raw storage capacity of about 1 gigabyte. Compliant benchmark implementations may also use one of the larger permissible database populations (e.g., 100 gigabytes), as defined in Clause 4.1.3). The performance metrics reported by TPC-H is called the TPC-H Composite Query-per-Hour Performance Metric (QphH@Size), and reflects multiple aspects of the capability of the system to process queries. These aspects include the selected database size against which the queries are executed, the query processing power when queries are submitted by a single stream, and the query throughput when queries are submitted by multiple concurrent users. The TPC-H Price/Performance metric is expressed as $/QphH@Size. To be compliant with the TPC-H standard, all references to TPC-H results for a given configuration must include all required reporting components (see Clause 5.4.6). The TPC believes that comparisons of TPC-H results measured against different database sizes are misleading and discourages such comparisons. The TPC-H database must be implemented using a commercially available database management system (DBMS), and the queries executed via an interface using dynamic SQL. The specification provides for variants of SQL, as implementers are not required to have implemented a specific SQL standard in full. Benchmarks results are highly dependent upon workload, specific application requirements, and systems design and implementation. Relative system performance will vary as a result of these and other factors. Therefore, TPC-H IBM Corporation TPC Benchmark H Full Disclosure Report March 23,

12 should not be used as a substitute for specific customer application benchmarking when critical capacity planning and/or product evaluation decisions are contemplated. IBM Corporation TPC Benchmark H Full Disclosure Report March 23,

13 1 General Items 1.1 Benchmark Sponsor A statement identifying the benchmark sponsor(s) and other participating companies must be provided. IBM Corporation sponsored this TPC-H benchmark. 1.2 Parameter Settings Settings must be provided for all customer-tunable parameters and options that have been changed from the defaults found in actual products, including but not limited to: Y Database tuning options Y Optimizer/Query execution options Y Query Processing tool/language configuration parameters Y Recovery/commit options Y Consistency/locking options Y Operating system and configuration parameters Y Configuration parameters and options for any other software component incorporated into the pricing structure Y Compiler optimization options. Appendix A contains the database and operating system parameters. 1.3 Configuration Diagrams Diagrams of both measured and priced configurations must be provided, accompanied by a description of the differences. This includes, but is not limited to: Y Number and type of processors Y Size of allocated memory and any specific mapping/partitioning of memory unique to the test and type of disk units (and controllers, if applicable) Y Number and type of disk units (and controllers, if applicable) Y Number of channels or bus connections to disk units, including their protocol type Y Number of LAN (e.g., Ethernet) connections, including routers, workstations, terminals, etc., that were physically used in the test or are incorporated into the pricing structure Y Type and run-time execution location of software components (e.g., DBMS, query processing tools/languages, middleware components, software drivers, etc.). The configuration diagram for the tested and priced system is provided on the following page. IBM Corporation TPC Benchmark H Full Disclosure Report March 23,

14 1.3.1 Measured Configuration xseries way 900MHz Pentium III Xeon with 2MB L2 Cache 4GB Memory 1 x Ethernet 10/100 PCI Onboard Interface Onboard Adaptec SCSI Interface 1 x 9.1GB 15K SCSI Drive 1 x ServeRAID-4H Ultra160 SCSI Adapter 1 x Netfinity EXP300 8 x 9.1GB 15K SCSI Drive RF Flat Files Database Log TempDB Log 1 x Netfinity EXP x 18.2GB 10K SCSI Drive Flat Files OS DBMS 4 x Mylex extremeraid 2000 Adapter 8 x Netfinity EXP x 9.1GB 15K SCSI Drive Database Data Backup Data TempDB Data The measured configuration for the xseries 350 contained: Four Intel Pentium III Xeon 900MHz processors, each with 2MB of L2 cache 4GB of memory One 10/100 Ethernet PCI onboard interface One onboard Adaptec SCSI interface Four Mylex extremeraid 2000 Adapters One ServeRAID-4H Ultra160 SCSI Adapter One hundred twenty-one (121) 9.1GB 15K Ultra160 SCSI Disk Drives Nine (9) EXP300 Storage Expansion Enclosures One (1) EXP200 Storage Expansion Enclosure (held flat files and was not priced) Ten (10) 18.2GB 10K Wide Ultra SCSI Disk Drives (held flat files and were not priced) Priced Configuration The priced and measured configurations were identical with one exception: Ten (10) 18.2GB Wide Ultra SCSI Disk Drives and one (1) EXP200 Storage Expansion Enclosure, which held flat files, were not priced. IBM Corporation TPC Benchmark H Full Disclosure Report March 23,

15 2 Clause 1: Logical Database Design Related Items 2.1 Database Table Definitions Listings must be provided for all table definition statements and all other statements used to set up the test and qualification databases. ( ) Appendix B contains the scripts that were used to set up the TPC-H test and qualification databases. 2.2 Database Organization The physical organization of tables and indexes within the test and qualification databases must be disclosed. If the column ordering of any table is different from that specified in Clause 1.4, it must be noted. Appendix B contains the scripts that were used to create the indexes on the test and qualification databases. 2.3 Horizontal Partitioning Horizontal partitioning of tables and rows in the test and qualification databases must be disclosed (see Clause 1.5.4). Horizontal partitioning was not used. 2.4 Replication Any replication of physical objects must be disclosed and must conform to the requirements of Clause 1.5.6). No replication was used. IBM Corporation TPC Benchmark H Full Disclosure Report March 23,

16 3 Clause 2: Queries and Update Functions Related Items 3.1 Query Language The query language used to implement the queries must be identified. SQL was the query language used. 3.2 Random Number Generation The method of verification for the random number generation must be described unless the supplied DBGEN and QGEN were used. The TPC-supplied DBGEN version and QGEN version were used to generate all database populations. 3.3 Substitution Parameters Generation The method used to generate values for substitution parameters must be disclosed. If QGEN is not used for this purpose, then the source code of any non-commercial tool used must be disclosed. If QGEN is used, the version number, release number, modification number and patch level of QGEN must be disclosed. The supplied QGEN version was used to generate the substitution parameters. 3.4 Query Text and Output Data from Database The executable query text used for query validation must be disclosed along with the corresponding output data generated during the execution of the query text against the quali fication database. If minor modifications (see Clause 2.2.3) have been applied to any functional query definitions or approved variants in order to obtain executable query text, these modifications must be disclosed and justified. The justification for a particular m inor query modification can apply collectively to all queries for which it has been used. The output data for the power and throughput tests must be made available electronically upon request. Appendix C contains the query text and query output. The following modifications were used: Y In Q1, Q4, Q5, Q6, Q10, Q12, Q14, Q15 and Q20, the dateadd function is used to perform date arithmetic. Y In Q7, Q8 and Q9, the datepart function is used to extract part of a date (e.g., YY ). Y In Q2, Q3, Q10, Q18 and Q21, the top function is used to restrict the number of output rows. 3.5 Query Substitution Parameters and Seeds Used All query substitution parameters used for all performance tests must be disclosed in tab ular format, along with the seeds used to generate these parameters. Appendix D contains the seed and query substitution parameters used. 3.6 Query Isolation Level The isolation level used to run the queries must be disclosed. If the isolation level does not map closely to one of the isolation levels defined in Clause 3.4, additional descriptive detail must be provided. The queries and transactions were run with isolation level 1. IBM Corporation TPC Benchmark H Full Disclosure Report March 23,

17 3.7 Refresh Function Implementation The details of how the refresh functions were implemented must be disclosed (including source code of any non-commercial program used). Appendix E contains the source code for the refresh functions. IBM Corporation TPC Benchmark H Full Disclosure Report March 23,

18 4 Clause 3: Database System Properties Related Items 4.1 ACID Properties The ACID (Atomicity, Consistency, Isolation and Durability) properties of transaction processing systems must be supported by the system under test during the timed portion of this benchmark. Since the TPC-H is not a transaction processing benchmark, the ACID properties must be evaluated outside the timed portion of the test. 4.2 Atomicity The system under test must guarantee that transactions are atomic; the system will either perform all individual operations on the data, or will assure that no partially completed operations leave any effects on the data Completed Transactions Perform the ACID transactions for a randomly selected set of input data and verify that the appropriat e rows have been changed in the ORDER, LINEITEM and HISTORY tables. The following steps were performed to verify the Atomicity of completed transactions. 1. The total price from the ORDER table and the extended price from the LINEITEM table were retrieved for a randomly selected order key. 2. The ACID Transaction was performed using the order key from step The ACID Transaction committed. 4. The total price from the ORDER table and the extended price from the LINEITEM table were retrieved for the same order key. It was verified that the appropriate rows had been changed Aborted Transactions Perform the ACID transactions for a randomly selected set of input data, submitting a ROLLBACK of the transaction for the COMMIT of the transaction. Verify that the appropriate rows have not been changed in the ORDER, LINEITEM and HISTORY tables. The following steps were performed to verify the Atomicity of the aborted ACID transaction: 1. The total price from the ORDER table and the extended price from the LINEITEM table were retrieved for a randomly selected order key. 2. The ACID Transaction was performed using the doer key from step 1. The transaction was stopped prior to the commit. 3. The ACID Transaction was ROLLED BACK.. 4. The total price from the ORDER table and the extended price from the LINEITEM table were retrieved for the same order key used in steps 1 and 2. It was verified that the appropriate rows had not been changed. 4.3 Consistency Consistency is the property of the application that requires any execution of transaction s to take the database from one consistent state to another Consistency Test Verify that ORDER and LINEITEM tables are initially consistent, submit the prescribed number of ACID Transactions with randomly selected input parameters, and reverify the consiste ncy of the ORDER and LINEITEM. The following steps were performed to verify consistency: 1. The consistency of the ORDER and LINEITEM tables was verified based on a sample of O_ORDERKEYs. 2. One hundred ACID Transactions were submitted from each of two execution streams. IBM Corporation TPC Benchmark H Full Disclosure Report March 23,

19 3. The consistency of the ORDER and LINEITEM tables was reverified. 4.4 Isolation Operations of concurrent transactions must yield results, which are indistinguishable from the results that would be obtained by forcing each transaction to be serially executed to completion in some order Read-Write Conflict with Commit Demonstrate isolation for the read-write conflict of a read-write transaction and a read-only transaction when the read-write transaction is committed. The following steps were performed to satisfy the test of isolation for a read-only and a read-write committed transaction: 1. An ACID Transaction was started for a randomly selected O_KEY, L_KEY and DELTA. The ACID Transaction was suspended prior to Commit. 2. An ACID query was started for the same O_KEY used in step 1. The ACID query blocked and did not see any uncommitted changes made by the ACID Transaction. 3. The ACID Transaction was resumed and committed. 4. The ACID query completed. It returned the data as committed by the ACID Transaction Read-Write Conflict with Rollback Demonstrate isolation for the read-write conflict of a read-write transaction and read-only transaction when the read-write transaction is rolled back. The following steps were performed to satisfy the test of isolation for read-only and a rolled back read-write transaction: 1. An ACID transaction was started for a randomly selected O_KEY, L_KEY and DELTA. The ACID Transaction was suspended prior to Rollback. 2. An ACID query was started for the same O_KEY used in step 1. The ACID query did not see any uncommitted changes made by the ACID Transaction. 3. The ACID Transaction was ROLLED BACK. 4. The ACID query completed Write-Write Conflict with Commit Demonstrate isolation for the write-write conflict of two update transactions when the first transaction is committed. The following steps were performed to verify isolation of two update transactions: 1. An ACID Transaction T1 was started for a randomly selected O_KEY, L_KEY and DELTA. The ACID transaction T1 was suspended prior to Commit. 2. Another ACID Transaction T2 was started using the same O_KEY and L_KEY and a randomly selected DELTA. 3. T2 waited. 4. The ACID transaction T1 was allowed to Commit and T2 completed. 5. It was verified that: T2.L_EXTENDEDPRICE T1.L_EXTENDEDPRICE + (DELTA1*(T1.L_EXTENDEDPRICE/T1.L_QUANTITY)) IBM Corporation TPC Benchmark H Full Disclosure Report March 23,

20 4.4.4 Write-Write Conflict with Rollback Demonstrates isolation for the write-write conflicts of two update transactions when the first transaction is rolled back. The following steps were performed to verify the isolation of two update transactions after the first one is rolled back: 1. An ACID Transaction T1 was started for a randomly selected O_KEY, L_KEY and DELTA. The ACID Transaction T1 was suspended prior to Rollback. 2. Another ACID Transaction T2 was started using the same O_KEY and L_KEY used in step 1 and a randomly selected DELTA. 3. T2 waited. 4. T1 was allowed to ROLLBACK and T2 completed. 5. It was verified that T2.L_EXTENDEDPRICE T1.L_EXTENDEDPRICE Concurrent Progress of Read and Write on Different Tables Demonstrate the ability of read and write transactions affecting different database tables to mak e progress concurrently. The following steps were performed: 1. An ACID Transaction T1 for a randomly selected O_KEY, L_KEY and DELTA. The ACID Transaction T1 was suspended prior to Commit. 2. Another ACID Transaction T2 was started using random values for PS_PARTKEY and PS_SUPPKEY. 3. T2 completed. 4. T1 completed and the appropriate rows in the ORDER, LINEITEM and HISTORY tables were changed Updates Not Indefinitely Delayed by Reads on Same Table Demonstrate that the continuous submission of arbitrary (read-only) queries against one or more tables of the database does not indefinitely delay update trans actions affecting those tables from making progress. The following steps were performed: 1. An ACID Transaction T1 was started, executing Q1 against the qualification database. The substitution parameter was chosen from the interval [ ] so that the query ran for a sufficient amount of time. 2. Before T1 completed, an ACID Transaction T2 was started using randomly selected values of O_KEY, L_KEY and DELTA. 3. T2 completed before T1 completed. 4. It was verified that the appropriate rows in the ORDER, LINEITEM and HISTORY tables were changed. 4.5 Durability The tested system must guarantee durability: the ability to preserve the effects of committed transactions and ensure database consistency after recovery from any one of the failures listed in Clause Failure of a Durable Medium Guarantee the database and committed updates are prese rved across a permanent irrecoverable failure of any single durable medium containing TPC-H database tables or recovery log tables. The database log was stored on a RAID-1 protected array of six physical drives. The tables for the database were stored on 14 RAID-0 arrays each containing 14 physical drives. Two backups of the database were taken. Each backup was spread across a distinct set of seven RAID-0 arrays. The tests were conducted on the qualification database. The steps performed are shown below. 1. The complete database was backed up twice to different sets of disk drives. IBM Corporation TPC Benchmark H Full Disclosure Report March 23,

21 2. Six streams of ACID transactions were started. Each stream executed a minimum of 100 transactions. 3. While the test was running, one of the disks from the database RAID-10 log was removed. 4. After it was determined that the test would still run with the loss of a log disk, one physical drive of a RAID-0 data volume was removed. 5. A checkpoint was issued to force a failure. 6. The six streams of ACID transactions failed and recorded their number of committed transaction in success files. 7. The database log was dumped to disk. 8. The data/backup file disk was swapped with another disk in the same array. The database data partitions containing the failed disk were deleted and recreated to ensure that the original database data on the drive was lost. 9. The failed physical drive affected only one of the two backup copies. A database restore was done using the unaffected copy. 10. A command was issued causing the database to run through its roll-forward recovery. 11. The counts in the success files and the HISTORY table count were compared and were found to match. 12. The database log disk was replaced and a rebuild function was initiated to restore the RAID-10 log array to its protected status. The rebuild completed successfully System Crash Guarantee the database and committed updates are preserved across an instantaneous interruption (system crash/system hang) in processing which requires the system to reboot to recover. 1. Six streams of ACID transactions were started. Each stream executed a minimum of 100 transactions. 2. While the streams of ACID transactions were running, the system was powered off. 3. When power was restored, the system rebooted and the database was restarted. 4. The database went through a recovery period. 5. The success file and the HISTORY table counts were compared and were found to match Memory Failure Guarantee the database and committed updates are preserved across failure of all or part of memory (loss of contents). See the previous section. IBM Corporation TPC Benchmark H Full Disclosure Report March 23,

22 5 Clause 4: Scaling and Database Population Related Items 5.1 Cardinality of Tables The cardinality (e.g., the number of rows) of each table of the test database, as it existed at the completion of the database load (see Clause 4.2.5), must be disclosed. Table Name Order Lineitem Customer Part Supplier Partsupp Nation Region Rows 150,000, ,037,902 15,000,000 20,000,000 1,000,000 80,000, Distribution of Tables and Logs The distribution of tables and logs across all media must be explicitly de scribed. The following series of tables show the distribution of tables and logs across all media. Controller Drives Partition* Size Use Adaptec 1-9.1GB C: D: 2.93GB (NTFS) 5.54GB (NTFS) OS, DBMS, Misc. (TPC-H Kit, etc.) ServeRAID-4H 6-9.1GB 2-9.1GB GB G: K: L: U: 25.41GB 14.65GB 2.29GB (NTFS) GB (NTFS) LogFile (RAID-10) TempLog (RAID-0) RF Flat Files (RAID-0) Flat Files (RAID-0) Mylex GB D:\devjp\lineitem_tables_file_1 D:\devjp\general_tables_file_1 D:\devjp\tempdev1 D:\devjp\backupdev1 D:\devjp\backup1gdev MB 11200MB 22400MB 47000MB (NTFS) 2000MB 10.32GB Lineitem File Group General File Group TempDB Backup (Qual. &Test DB) Unused Unused GB D:\devjp\lineitem_tables_file_2 D:\devjp\general_tables_file_2 D:\devjp\tempdev2 D:\devjp\backupdev2 D:\devjp\backup1gdev MB 11200MB 22400MB 47000MB (NTFS) 2000MB 10.32GB Lineitem File Group General File Group TempDB Backup (Qual.&Test DB) Unused Unused IBM Corporation TPC Benchmark H Full Disclosure Report March 23,

23 Controller Drives Partition* Size Use Mylex GB D:\devjp\lineitem_tables_file_3 D:\devjp\general_tables_file_3 D:\devjp\tempdev3 D:\devjp\backupdev3 D:\devjp\backup1gdev MB 11200MB 22400MB 47000MB (NTFS) 2000MB 10.32GB Lineitem File Group General File Group TempDB Backup (Qual.&Test DB) Unused Unused GB D:\devjp\lineitem_tables_file_4 D:\devjp\general_tables_file_4 D:\devjp\tempdev4 D:\devjp\backupdev4 D:\devjp\backup1gdev MB 11200MB 22400MB 47000MB (NTFS) 2000MB 10.32GB Lineitem File Group General File Group TempDB Backup (Qual.&Test DB) Unused Unused Mylex GB D:\devjp\lineitem_tables_file_5 D:\devjp\general_tables_file_5 D:\devjp\tempdev5 D:\devjp\backupdev5 D:\devjp\backup1gdev MB 11200MB 22400MB 47000MB (NTFS) 2000MB 10.32GB Lineitem File Group General File Group TempDB Backup (Qual.&Test DB) Unused Unused GB D:\devjp\lineitem_tables_file_6 D:\devjp\general_tables_file_6 D:\devjp\tempdev6 D:\devjp\backupdev6 D:\devjp\backup1gdev MB 11200MB 22400MB 47000MB (NTFS) 2000MB 10.32GB Lineitem File Group General File Group TempDB Backup (Qual.&Test DB) Unused Unused Mylex GB D:\devjp\lineitem_tables_file_7 D:\devjp\general_tables_file_7 D:\devjp\tempdev7 D:\devjp\backupdev MB 11200MB 22400MB 58.17GB (NTFS) Lineitem File Group General File Group TempDB Backup (Test DB) GB D:\devjp\lineitem_tables_file_8 D:\devjp\general_tables_file_8 D:\devjp\tempdev8 D:\devjp\backupdev8, X: 28000MB 11200MB 22400MB 58.17GB (NTFS) Lineitem File Group General File Group TempDB Backup (Test DB), Page File * The physical drives that do not have drive letters are assigned Windows 2000 junction points. The priced configuration used 121 disks. An additional 10 disks were used to store the 100GB database files and were not priced. 5.3 Database Partition / Replication Mapping The mapping of database partitions/replications must be explicitly described. Database partitioning/replication was not used. 5.4 RAID Implementation Implementations may use some form of RAID to ensure high availab ility. If used for data, auxiliary storage (e.g., indexes) or temporary space, the level of RAID must be disclosed for each device. RAID-10 was used for log disks. RAID-0 was used for all other database disks and for the TempLog. 5.5 DBGEN Modifications IBM Corporation TPC Benchmark H Full Disclosure Report March 23,

24 Any modifications to the DBGEN (see Clause 4.2.1) source code must be disclosed. In the event that a program other than DBGEN was used to populate the database, it must be disclosed in its entirety. The standard distribution DBGEN version was used for database population. No modifications were made. 5.6 Database Load Time The database load time for the test database (see Clause 4.3) must be disclosed. See the Executive Summary at the beginning of this report. 5.7 Data Storage Ratio The data storage ratio must be disclosed. It is computed as the ratio between the total amount of priced disk spac e and the chosen test database size as defined in Clause The calculation of the data storage ratio is shown in the following table. Disk Type Number of Disks Formatted Space per Disk Total Disk Space Scale Factor Storage Ratio 9.1GB 15K Ultra160 SCSI Drive GB 1, The data storage ratio is 10.25, derived by dividing GB by the database size of 100GB. IBM Corporation TPC Benchmark H Full Disclosure Report March 23,

25 5.8 Database Load Mechanism Details and Illustration The details of the database load must be disclosed, including a block diagram illustrating the overall process. Disclosure of the load procedure includes all steps. scripts, input and configuration files required to completely reproduce the test and qualification databases. Flat files for each of the tables were created using DBGEN. The tables were loaded as depicted in Figure 4-1. Figure 4-1. Database Load Procedure Create Flat Data Files Create Database Configure for Load Create and Load Tables database load timing period Create Indexes Create Statistics Install Refresh functions First Backup Second Backup End of Load 5.9 Qualification Database Configuration Any differences between the configuration of the qual ification database and the test database must be dis closed. The qualification database used identical scripts and disk structure to create and load the data with adjustments for size difference. IBM Corporation TPC Benchmark H Full Disclosure Report March 23,

26 6 Clause 5: Performance Metrics and Execution Rules Related Items 6.1 System Activity between Load and Performance Tests Any system activity on the SUT that takes place between the conclusion of the load test and the beg inning of the performance test must be fully disclosed. The database system was restarted. 6.2 Steps in the Power Test The details of the steps followed to implement the power test (e.g., system reboot, database restart) must be disclosed. The following steps were used to implement the power test: 1. RF1 Refresh Transaction 2. Stream 00 Execution 3. RF2 Refresh Transaction 6.3 Timing Intervals for Each Query and Refresh Function The timing intervals for each query of the measured set and for both update functions must be reported for the power test. See the Numerical Quantities Summary in the Executive Summary at the beginning of this report. 6.4 Number of Streams for the Throughput Test The number of execution streams used for the throughput test must be disclosed. Five streams were used for the throughput test. 6.5 Start and End Date/Times for Each Query Stream The start time and finish time for each query execution stream must be reported for the throughput test. See the Numerical Quantities Summary in the Executive Summary at the beginning of this report. 6.6 Total Elapsed Time for the Measurement Interval The total elapsed time for the measurement interval must be reported for the throughput test. See the Numerical Quantities Summary in the Executive Summary at the beginning of this report Refresh Function Start Date/Time and Finish Date/Time The start time and finish time for each update function in the update stream mu st be reported for the throughput test. See the Numerical Quantities Summary in the Executive Summary at the beginning of this report. IBM Corporation TPC Benchmark H Full Disclosure Report March 23,

27 6.8 Timing Intervals for Each Query and Each Refresh Function for Each Stream The timing intervals for each query of each stream and for each update function must be reported for the throughput test. See the Numerical Quantities Summary in the Executive Summary at the beginning of this report. 6.9 Performance Metrics The computed performance metrics, related numerical quantities, and the price/performance metric must be reported. See the Numerical Quantities Summary in the Executive Summary at the beginning of this report Performance Metric and Numerical Quantities from Both Runs The performance metric and numerical quantities from both runs must be disclosed. Two consecutive runs of the TPC-H benchmark were performed. The following table contains the results for both runs. 100GB 100GB 100GB Run Run % Difference 0.86% 0.56% 0.72% 6.11 System Activity between Tests Any activity on the SUT that takes place between the conclusion of Run1 and the beginning of Run2 must be disclosed. A database checkpoint was issued. The database server was restarted between runs. IBM Corporation TPC Benchmark H Full Disclosure Report March 23,

28 7 Clause 6: SUT and Driver Implementation Related Items 7.1 Driver A detailed textual description of how the driver performs its functions, how its various components interact and any product functionality or environmental setting on which it relies must be provided. All related source code, scripts and configurations must be disclosed. The information provided should be sufficient for an independent reconstruction of the driver. Two scripts were used from the Microsoft tool named StepMaster. The first one is Data Load, which was used to create and load the database as well as the backup. The second one is Power/Throughput Run, which was used to run the Power and Throughput tests. The source of this program is contained in Appendix F. The power test is invoked by calling tpcdbatch with the stream number 0 specified, an indication that the update functions must be run, and the SQL file that contains the power stream queries. 7.2 Implementation-Specific Layer If an implementation-specific layer is used, then a detailed description of how it performs its functions must be supplied, including any related source code or scripts. This description should allow an independent reconstruction of the implementation-specific layer. The StepMaster program is used to control and track the execution of queries. The scripts are listed in Appendix F. The following steps are performed to accomplish the Power and Throughput runs Power Run 1. Execute 16 concurrent RF1 processes, each of which will apply a segment of an update set generated by DBGEN. Each process submits multiple transactions, where a transaction spans a set of orders and their associated line items. 2. Execute the Stream 0 queries in the prescribed order. 3. Execute 16 concurrent RF2 processes, each of which will apply a segment of an update set generated by DBGEN. Each process submits multiple transactions, where a transaction spans a set of orders and their associated line items Throughput Run 1. Execute five concurrent query streams. Each stream executes queries in the prescribed order for the appropriate Stream Id (1-5). Upon completion of each stream, a semaphore is set to indicate completion. 2. Execute five consecutive RF1/RF2 transactions against ascending Update sets produced by DBGEN. The first RF1 waits on a semaphore prior to beginning its insert operations. Each step is timed by the script. The timing information is stored in an output file in StepMaster for later analysis. IBM Corporation TPC Benchmark H Full Disclosure Report March 23,

29 8 Clause 7: Pricing Related Items 8.1 Hardware and Software Components A detailed list of the hardware and software used in the priced system must be reported. Each item must have a vendor part number, description and release/revision level, and either general availability status or committed delivery date. If package-pricing is used, contents of the package must be disclosed. Pricing source(s) and effective date(s) must also be reported. A detailed list of all hardware and software, including the 5-year price, is provided in the Executive Summary at the front of this report. The price quotations are included in Appendix H. 8.2 Five-Year Cost of System Configuration The total 5-year price of the entire configuration must be reported, including hardware, software and maintenance charges. Separate component pricing is recommended. The basis of all discounts must be disclosed. A detailed list of all hardware and software, including the 5-year price, is provided in the Executive Summary at the front of this report. The price quotations are included in Appendix H. 8.3 Availability Dates The committed delivery date for general availability (availability date) of products used in the price calculations must be reported. When the priced system includes products with different availability dates, availability date reported on the Executive Summary must be the date by which all components are committed to being available. The Full Disclosure Report must report availability dates individually for at least each of the categories for which a pricing subtotal must be provided (see Clause ). The xseries 350 server will be generally available April 13, All other hardware, operating system and database software used in this benchmark were generally available at the time of publication. The following ServeRAID-4H software will be available May 31, 2001 and can be downloaded from IBMs Web site: Device driver version Firmware version BIOS version The total solution availability date is May 31, Country-Specific Pricing Additional Clause 7 related items may be included in the Full Disclosure R eport for each country-specific priced configuration. Country-specific pricing is subject to Clause The configuration is priced for the United States of America. IBM Corporation TPC Benchmark H Full Disclosure Report March 23,

30 Clause 9: Audit Related Items 9.1 Auditor s Report The auditor s agency name, address, phone number, and Attestation letter with a brief audit summary report indicating compliance must be included in the Full Disclosure Report. A statement should be included specifying who to contact in order to obtain further information regarding the audit process. This implementation of the TPC Benchmark H was audited by Brad Askins of InfoSizing, Inc. Further information may be obtained from: InfoSizing, Inc North Franklin Street Colorado Springs, CO Phone: Fax: IBM Corporation TPC Benchmark H Full Disclosure Report March 23,

31 Benchmark Sponsors: Richard Laviano Mgr. Server Systems Performance IBM Personal Systems Group 3039 Cornwallis Road Research Triangle Park, NC September 25, 2001 I verified the TPC Benchmark H performance of the following configuration: Platform: IBM ^ xseries 350 Database Manager: Microsoft SQL Server 2000 Enterprise Edition Operating System: Microsoft Windows 2000 Advanced Server The results were: CPU (Speed) 4 x Intel Pentium III Xeon (900 MHz) Memory Disks QphH@100GB IBM ^ xseries 350 2MB Cache/cpu 4 GB Main 121 x 9.1 GB 1,169.7 In my opinion, this performance result was produced in compliance with the TPC s requirements for the benchmark. The following verification items were given special attention: The database records were defined with the proper layout and size The database population was generated using DBGEN The database was properly scaled to 100GB and populated accordingly The compliance of the database auxiliary data structures was verified The database load time was correctly measured and reported 1373 North Franklin Street Colorado Springs, CO Office: 719/ Fax: 719/