Dialog (interactive) data input. Reporting. Printing processing

Transcription

1 Tutorials, D. Prior Research Note 24 February 2003 Who Sets the Pace in the SAP Performance 'Olympics'? SAP and its hardware vendors use many different application performance benchmarks. But records for these R/3 benchmark tests do not stand for long and vendors favor different benchmarks, so comparing results is hard. Core Topics Enterprise Management: Database Management and Administration ERP II, Supply Chain & Manufacturing: ERP II - Strategies, Applications and Technologies Key Issues How will ERP II technology and architectures evolve? What strategies and best practices promote effective database management? Strategic Planning Assumption Through 2007, end-user response time will remain one of the top five critical success factors for an ERP or ERP II implementation (0.8 probability). Performance benchmarks are standard programs and reference data that compare the performance of different computer systems. SAP has been working with its hardware partners since 1993 to develop a range of standard application performance benchmarks to: Characterize the scalability of its business application products Calibrate the system sizing process for individual customer projects Measure the resource footprint the amount of hardware needed for the same level of performance of new releases of SAP products Measure the SAP performance of new hardware In "The SAP Performance 'Olympics': Who Holds Gold for R/3?" TU , we explained the theory and the practice of SAP standard application benchmarks. Since then, the records for these R/3 benchmark results have moved on considerably. At the same time, the diversity of the SAP benchmark types has increased. SAP Benchmark Types: There are five types of SAP business workload: Dialog (interactive) data input Batch (background) data input/output Reporting Printing processing Gateway (system/system communication) processing Gartner Reproduction of this publication in any form without prior written permission is forbidden. The information contained herein has been obtained from sources believed to be reliable. Gartner disclaims all warranties as to the accuracy, completeness or adequacy of such information. Gartner shall have no liability for errors, omissions or inadequacies in the information contained herein or for interpretations thereof. The reader assumes sole responsibility for the selection of these materials to achieve its intended results. The opinions expressed herein are subject to change without notice.

2 SAP has developed a range of benchmarks to test the business performance of various mysap components using its two-tier or three-tier (combined database and application server) architecture. SAP's R/3 Sales and Distribution (SD) logistics benchmark has been the most visible, with more than 210 certified results to date. It tests dialog workloads of standard SD transactions, but it excludes credit limit checks, which impose additional loads. The SD benchmark also includes simple online pricing and available to promise (ATP) checking. Note 1 The SAP Benchmark Record Definitions Explanation of column headings in Figure 1 Benchmark Type: Sales & distribution (SD) or assemble to order (ATO). Two or Three Tier: Two tier means combined database and application servers; three tier means separate server layers. Benchmark Users: The maximum number of standard users running the SD or ATO transaction benchmark scripts defined by SAP, with an average dialog response time of less than two seconds. In 1999, SAP introduced the R/3 assemble to order (ATO) benchmark to simulate the standard dialog processing needed for configurable manufacturing environments, but it also tests some printing and gateway processing. There are SAP benchmarks for the R/3 FI, MM and PP modules, the Business Information Warehouse (BW), Advanced Planner and Optimizer (APO) and Online Store components, plus parts of the retail, utilities and banking industry solutions. However, there are no standard benchmarks for important mysap solutions such as customer relationship management (CRM), supply chain management (SCM), product life cycle management (PLM), supplier relationship management (SRM) and the Enterprise Portal, or the majority of the industry solutions. SAP R/3 Benchmarks 'Olympic' Records in 2003: Figure 1 summarizes the highest R/3 performance results that have been certified for the SD and ATO benchmarks on the Windows, Unix, AS/400 and Linux platform types. Figure 1 column definitions are provided in Note 1. Since the database server constrains the overall performance of a SAP system, Figure 1 includes only the database server configuration. More detail is available for each of these results ( SAPS: A long-standing performance definition developed by SAP where 100 SAPS equal 2,000 SD order line items. These have been fully processed in one hour by a computer system at 100 percent CPU utilization. This requires 6,000 dialog steps, 2,000 actual postings and 2,400 complete R/3 transactions. This rating is release dependent. Business Items: The number of transaction items, for example, order lines or complete orders depending on the benchmark type. 24 February

3 Benchmark Type Two or Three Tier Figure 1 SAP Benchmark Records for SD and ATO R/3 OLTP Performance: February 2003 Benchmark Users Business Items/Hour Date R/3 Release DBS Platform Type Hardware Vendor Database Server Database SD: e-@ction ES SD 135,670 Nov 4.6C W AS LE Unisys Aries 130/16, MS SQL Server Sales Orders (6,780 SAPs) (order lines/hour) 2002 Itanium II, 1.0GHz ,000 SD 813,000 Dec 4.6C Unix Sun Sun Fire 15000/104, Oracle 9i (40,650 SAPS) (order lines/hour) 2002 UltraSPARC III, 1200MHz 2 1,325 SD 135,670 Jun 4.6C AS/400 IBM eserver iseries /24 DB2 for iseries (6,780 SAPS) (order lines/hour) 2001 Power PC, 600MHz V5R1M0 2 IBM xseries SD 69,330 Sep 4.6C Linux Model RX/8, (3,470 SAPS) (order lines/hour) 2002 SuSe Pentium III Xeon, 2.0GHz DB2 UDB V7.2 ATO: 2 1,610 Feb 4.6B W2000 Hewlett- NetServer LXr8500/4, MS SQL Server Assembly (orders/hour) 2001 Packard Pentium III Xeon, 700MHz Orders 2 34,260 May 4.6B Unix Fujitsu Primepower 2000/128, Oracle (orders/hour) 2001 Siemens SPARC 64, 560MHz 2 2,390 Jun 4.0B AS/400 IBM AS/400e DB2 for V4R4 Model S /12, (orders/hour) 1999 Power PC, 262MHz SD: 26,000 SD 2,606,000 Feb 4.6C W2000 Unisys e-@ction ES7000/32, MS SQL Server 3 Sales Orders (130,300 SAPS) (order lines/hour) 2002 Pentium III Xeon, 900MHz ,528 SD 4,799,330 Sep 4.6C Unix IBM eserver Model p690/32, DB2 V8.1 (239,970 SAPS) (order lines/hour) 2002 Power4, 1.6GHz 672, ,651 SD Aug 3.0F AS/400 IBM AS/400e DB2 (33,600 SAPS) (order lines/hour) 1998 Model S /12, for OS/400 Power PC, 262MHz V4R3 3 2,500 SD 250,330 Nov 4.6C Linux Fujitsu PRIMERGY H400/4, SAP DB 7.3 (12,520 SAPS) (order lines/hour) 2002 Siemens Pentium III Xeon, 900MHz ATO: 3 9,130 Apr 4.0B NT4 Hewlett- NetServer LXr8000/4, Oracle Assembly 1999 Orders (orders/hour) Packard Pentium III Xeon, 500MHz 3 144,090 Dec 4.6C Unix Hewlett- HP9000 SuperDome/64 Oracle 9i (orders/hour) 2001 Packard PA-RISC 8700, 750MHz Source: Gartner Research Interpreting Benchmark Results: Figure 1 shows that the SD benchmark includes a "benchmark user" count and a useful SAPS standard performance rating that is widely used in the SAP sizing process. We do not believe that the benchmark user counts are very meaningful because "users" do not equate to real-world users. However, in all cases, we believe the total throughput figure in the "business items/hour" column is a very valuable indicator for sizing the tested application release and SAP workload type. SAP benchmarks typically focus on one workload type, whereas real-world SAP production systems must balance all five workload types. On Unix benchmarks, Oracle remains the preferred database for hardware vendors, with the notable exception of IBM, which prefers DB2. When comparing benchmark results, it should be remembered that there is a large variation in CPU technology, CPU clock speed and the number of CPUs used in each database server. However, Figure 1 can still be used to compare these benchmark throughputs per CPU to gauge relative performance across the various servers and platform types. The business items/hour, 24 February

4 application component release, CPU utilization and response time data from a SAP-certified benchmark are the most useful sizing data for determining the upper limits of dialog or batch throughput. SAP Benchmark Trends Breaking Different Records: The highest SD three-tier record was held for seven months from February 2002, but it was on Windows and not Unix. The 32-way Unisys result of 26,000 SD (benchmark users) was impressive and was extensively used by Microsoft and Wintel vendors. Wintel vendors are trying desperately to prove that they are ready for large-scale enterprise resource planning (ERP). The current SD three-tier record is now 47,528. This is on Unix, with IBM's p way machine and was a huge, 86 percent increase on IBM's previous Unix record. Hewlett-Packard (HP) and Sun Microsystems, the two other leading Unix vendors, have lost interest in the three-tier SD benchmark. HP is refocusing on benchmarks such as ATO and APO, which support its supply chain interests. In December 2002, after a 30-month absence, Sun reappeared in the Figure 1 SD record table by taking the two-tier SD record with its 104-way Sun Fire The main SD benchmark trend is for the two-tier approach, as it is much easier and less costly to perform. It appears that the three leading Unix vendors want to avoid direct SAP performance comparison. This makes it harder for enterprises to gauge relative SAP performance of Unix hardware configurations for large-scale SAP projects. As a result, enterprises have little data with which to challenge the sizing configurations from HP and Sun. The xseries (Windows) and pseries (Unix) divisions within IBM have been very active with SAP SD benchmarks, but the opposite is true for IBM's proprietary iseries (AS/400) and zseries (OS/390) divisions. The zseries mainframe is a very appealing SAP database server, especially in the multidatabase world of mysap.com. The absence of published SAP SD benchmark data can only diminish the otherwise great appeal of IBM mainframes for SAP database servers. This absence is because of the higher cost per transaction of mainframes. Since June 2001, the three-tier SD benchmark record values have grown for Windows platforms by almost four times and for Unix platforms by nearly three times. Linux has also appeared as a new SAP platform. Although there are only 1,100 SAP customer database server installations on Linux, it has tremendous potential as a highly cost-effective SAP application server or nonproduction system. Fujitsu Siemens Computers proved this in November 2000, with 23,000 SD for a 64-way 24 February

5 Solaris database server, plus 160 four-way Intel-based Primergy application servers. In February 2002, IBM achieved 520 SD. This was from an eightway Pentium III Xeon two-tier server running Windows With the same hardware and database, IBM achieved 575 SD on a Linux operating system, which is another SAP performance proof point for Linux. SAP's Web Application Server technology is developed on Linux. SAP's 64-bit kernel will become available for Windows and Linux platforms during 2003 (0.8 probability). Future SAP Benchmark Trends: In June 2002, SAP certified a notable three-tier, 12,000 SD parallel benchmark by HP using four AlphaServer ES45 four-way database servers, a high-speed memory interconnect and Tru64 Unix, all clustered with Oracle's 9i Real Application Clusters (RAC) parallel database management system (DBMS). This four-node benchmark demonstrated 75 percent scalability over a single node in terms of order line items per hour throughput. The additional availability and scalability that 9i RAC could potentially offer would be extremely useful in large-scale SAP production systems. However, the Oracle 9i RAC DBMS has yet to prove itself in realworld systems. SAP has yet to specify dates for customer availability of SAP applications with the Oracle 9i RAC parallel DBMS. Gartner expects an increase in heterogeneous, three-tier benchmarks featuring Unix database servers and Linux application servers because of the excellent price/performance ratio of Linux servers. HP's Itanium chip technology is also expected to greatly improve the price/performance of SAP on low-end Linux, Windows and Unix server configurations. Key Facts: SAP benchmark records provide useful guidelines for sizing SAP production systems (see "Sizing SAP Servers: Art or Science?" TU ). There are 17 types of SAP online transaction processing (OLTP) performance benchmarks; the most widely used are SD and ATO. Five types of business workloads must be balanced in a realworld R/3 production system: Dialog (interactive) data input, batch data input/output, reporting, printing and gateway (system/system communication) processing. The business items per hour benchmark data should be used as a business workload indicator. 24 February

6 Acronym Key APO Advanced Planner and Optimizer ATO Assemble to order DB2 Database 2 DBMS Database management system FI Financials MM Materials management PP Production planning RAC Real Application Clusters SD Sales and Distribution Bottom Line: SAP and its hardware partners have invested heavily in benchmarking the business performance of R/3. However, our advice for enterprises that need to size large R/3 systems near the top end of a platform's performance is to always interpret benchmark results carefully. The business items per hour, application component release, CPU utilization and response time data from a SAP-certified benchmark are the most useful sizing data in determining the upper limit for dialog or batch throughput. Real-world SAP production systems must balance all business workload types and be correctly sized to handle realistic peak business workloads. 24 February