Load Testing and Monitoring Oracle Real Application Clusters (RAC)

Transcription

1 Load Testing and Monitoring Oracle Real Application Clusters (RAC) White Paper written by Claudia Fernandez and Bernard Farrell Quest Software, Inc.

2 Copyright Quest Software, Inc All rights reserved. This guide contains proprietary information, which is protected by copyright. The software described in this guide is furnished under a software license or nondisclosure agreement. This software may be used or copied only in accordance with the terms of the applicable agreement. No part of this guide may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying and recording for any purpose other than the purchaser's personal use without the written permission of Quest Software, Inc. WARRANTY The information contained in this document is subject to change without notice. Quest Software makes no warranty of any kind with respect to this information. QUEST SOFTWARE SPECIFICALLY DISCLAIMS THE IMPLIED WARRANTY OF THE MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. Quest Software shall not be liable for any direct, indirect, incidental, consequential, or other damage alleged in connection with the furnishing or use of this information. TRADEMARKS All trademarks and registered trademarks used in this guide are property of their respective owners. World Headquarters 5 Polaris Way Aliso Viejo, CA info@quest.com U.S. and Canada: Please refer to our Web site for regional and international office information. Updated June 6, 2006 WPD_LoadTestingOracleRAC_081106_NH

3 CONTENTS INTRODUCTION...1 WHAT IS RAC DATABASE LOAD TESTING?...2 RAC LOAD TESTING OBJECTIVES...3 ENSURING CLUSTER STABILITY USING STANDARD BENCHMARKS... 4 VERIFYING RAC SYSTEM SCALING... 5 PROPERLY TUNING COMPONENTS... 5 RAC TESTING BEFORE IMPLEMENTING A MAJOR ORACLE RELEASE... 5 RAC TESTING SOLUTIONS...6 BENCHMARK FACTORY FOR DATABASES... 6 SPOTLIGHT ON RAC... 7 IMPLEMENTING THE RAC LOAD TESTING PROCESS RECORDING PRODUCTION ACTIVITY AND COLLECTING SQL WAIT EVENTS RUNNING THE RAC LOAD TEST WITH BENCHMARK FACTORY FOR DATABASES... 8 a. Using Predefined Load Testing Criteria against the Various Instances... 9 b. RAC Testing Using the Standard Load Balancing Criteria MONITORING RAC WITH QUEST SPOTLIGHT ON RAC THE UNIQUE CAPABILITIES OF BENCHMARK FACTORY FOR DATABASES AND SPOTLIGHT ON RAC...19 CONCLUSION...20 ABOUT THE AUTHOR...21 ABOUT QUEST SOFTWARE, INC CONTACTING QUEST SOFTWARE CONTACTING QUEST SUPPORT i

4 White Paper INTRODUCTION Oracle Real Application Clusters (RAC) combines several independent servers/nodes into one database system. As a clustered database solution, RAC provides two major functions: scalability and availability to a business continuum. Scalability is a relative term, based on a simple rule that as the number of users accessing the system increases, the RAC configuration should be able to handle the increased activity. Availability is the ability of the system to provide continuous service when one or more of the components in the cluster fails. To ensure that RAC delivers the desired levels of scalability and availability, the cluster must be able to load balance across all nodes, so that when one or more nodes in the cluster fails, users are not affected and availability is maintained. When more capacity is required, additional nodes can be added with minimal effort. Organizations spending time and money setting up RAC, however, must ensure their systems have the stability to withstand user environment demands when adding nodes. This white paper presents a load testing methodology used to implement, manage and perform full diagnostic capabilities of Oracle Real Application Clusters, in order to obtain optimum user service times regardless of the demands placed on the RAC environment. 1

5 Load Testing and Monitoring Oracle Real Application Clusters (RAC) WHAT IS RAC DATABASE LOAD TESTING? RAC database load testing determines system performance at a variety of usage levels, so DBAs can ensure that the two primary goals of a RAC implementation scalability and availability are met. Improper configuration or testing methods prevent many companies from attaining peak efficiency with their databases, leaving customers and end users frustrated by poor application performance. Load testing is essential to ensure that a RAC system will meet the required demands created by a change in database workload or configuration. 2

6 White Paper RAC LOAD TESTING OBJECTIVES Organizations investing time and money setting up RAC need to ensure scalability and availability. The main objectives during the RAC load testing process are: Ensuring the correct cluster configuration. Verifying that the hardware configuration, including the operating system, will scale as advertised by the manufacturer. Because every hardware combination is not the same, there are different permutations and combinations of components. Their individual performance characteristics and the combined performance characteristics may not be the same. Verifying that the application will scale on the cluster. Even when the application originally worked on a single (non-rac) instance, there is no guarantee it will scale on RAC. Properly tuned components. 3

7 Load Testing and Monitoring Oracle Real Application Clusters (RAC) Ensuring Cluster Stability Using Standard Benchmarks Load testing RAC is the best way to ensure database stability. Standard benchmarks, such as the AS3AP and TPC-C, can be utilized to create workloads. Before starting any testing, you will need to shut down all instances in the cluster except one. By load testing the one instance, you are in effect tuning the operating system setting. That is, you are ensuring that the one node is configured correctly before adding the overhead of the cluster ware and other cluster-related components, such as the interconnect. After testing the first instance, the configuration changes from that test are automatically copied to the second instance, ensuring that the second node in the cluster is tuned. Once both nodes have been tuned, you can run load tests across all nodes in the cluster, each with a different goal or target. The following list outlines what to look for during load testing: Interconnect traffic on the cluster: This is an important test. The interconnect is the primary hardware layer that provides data transfer between all nodes in the cluster. Problems with the interconnect configuration (e.g., the parameters/buffer sizes are not correctly set) can be identified. As more data is shared between the instances, the performance of the cluster degrades if the cluster is not properly tuned. Failover component performance: One of the primary reasons for implementing RAC is to improve server availability. A load test can measure performance failover and availability of component clusters. Failures can include interconnect, fibre channels to the Storage Area Networks (SAN), nodes and instances. General load tests to tune the Oracle-related parameters: An industry-standard benchmark, such as the TPC-C, generates load tests against both instances in the cluster, allowing tuning of Oracle-related parameters. After running a load test and obtaining acceptable performance, you can save these results/parameters as a baseline to establish milestones in the load testing methodology. This establishes a cluster that is stable and tuned, without incorporating any of the customer application layers. 4

8 White Paper Verifying RAC System Scaling After you load test the instances, it is important to see how the application data scales in the RAC environment. This is an area where RAC implementation often fails. Many database professionals think that RAC will simply migrate to their application, without testing scalability. To test scalability, you can parse an Oracle SQL trace file generated using a event, and then generate a load against the database instances. This isolates the database tier from the application tier, and tests all the queries that users would normally execute via the application. If the same queries do not perform well when executed through the application, it clearly indicates that the application is the source of the bottleneck and not the database, as one might suspect. Properly Tuning Components After running a load test and determining a RAC configuration, users must locate SQL statements in production that need to be optimized to obtain acceptable performance. This completes the RAC testing process, ensuring that all facets of the database have been tested to provide optimum user satisfaction. RAC Testing Before Implementing a Major Oracle Release It is recommended that before implementing a major Oracle release, such as Oracle10g, or a major configuration change such as RAC, that you simulate your production workload on the new database environment. This allows you to gather testing data on what to expect in terms of database performance and scalability. Examples include the difference in performance between Oracle Database 9i and Oracle Database 10g, Oracle non-rac and Oracle RAC configurations, and using a different number of RAC nodes. 5

9 Load Testing and Monitoring Oracle Real Application Clusters (RAC) RAC TESTING SOLUTIONS Quest Software provides two solutions for RAC load testing and diagnostics: Benchmark Factory for Databases Spotlight on RAC Benchmark Factory for Databases Quest Software s Benchmark Factory for Databases places enormous stress on a RAC system stress that is difficult to achieve in a standard testing environment. By identifying system capacity and performance bottlenecks before they occur, Benchmark Factory facilitates proactive testing, which reduces downtime, development costs and potential loss of revenue. Benchmark Factory provides the ability to: Distribute load tests over RAC instances. Determine the right number of RAC nodes. Simulate thousands of concurrent users with a minimal amount of hardware, providing scalability to simulate large numbers of users. Generate loads using industry-standard benchmarks (AS3AP, TPC-B, TPC- C, TPC-D and Scalable Hardware). Generate loads by importing and replaying Oracle SQL trace files, and other types of files with SQL scripts. Control agents installed on client machines to spawn multiple virtual-user sessions, allowing the workload to be representative of a real-world user environment. Determine database throughput and capacity with reliable data points. View real-time testing results, providing immediate feedback of database performance on individual RAC nodes and allowing an accurate assessment of a critical database system. Compare load performance metrics across different databases and Oracle RAC nodes. 6

10 White Paper Spotlight on RAC Spotlight on RAC provides full diagnostic capability for an Oracle RAC environment. Spotlight on RAC takes RAC diagnostics to a whole new level by collecting performance criteria on each instance in the cluster, and then doing appropriate calculations to provide RAC-wide diagnostic information. This allows you to quickly discover any performance bottleneck, whether at the node, cluster or interconnect level, in real time. Spotlight identifies and diagnoses thousands of performance issues, whether it's a specific user, resource-intensive SQL transaction, I/O bottleneck, lock wait or other exact source. Spotlight on RAC answers many questions related to RAC performance such as: How is each of the instances in my cluster performing? How is the database as a whole performing? Is my database experiencing cluster-related contention? 7

11 Load Testing and Monitoring Oracle Real Application Clusters (RAC) IMPLEMENTING THE RAC LOAD TESTING PROCESS Testing a RAC system using Benchmark Factory for Databases and Spotlight on RAC is straightforward and requires only a minimal amount of setup. The process consists of the following steps: 1. Recording production activity with an Oracle trace file and collecting SQL wait events. 2. This allows you to simulate exact production activity, including session concurrency, latencies and bind variable values. 3. Running the RAC load test with Benchmark Factory for Databases. 4. Monitoring the RAC systems under load with Spotlight on RAC. 1. Recording Production Activity and Collecting SQL Wait Events Enabling tracing in Oracle allows you to record production activity, including concurrency between sessions, session activity and latencies that include bind variables. To activate tracing in Oracle, use the following command: ALTER SYSTEM SET EVENTS trace name context forever, level 4 Quest Central for Oracle offers a Visual Trace utility to set up Oracle trace files. 2. Running the RAC Load Test with Benchmark Factory for Databases Testing using Benchmark Factory for Databases can be approached from two angles: a) Using a predefined load against each of the instances in the cluster b) Using the standard load balancing criteria provided by Oracle to test each of the instances in the cluster 8

12 a. Using Predefined Load Testing Criteria against the Various Instances White Paper Under this approach, the first step in the Benchmark Factory RAC load testing process is to set up the Oracle tnsnames.ora file. i. Setting up the tnsnames.ora file to load test the RAC system To test a RAC system using Benchmark Factory, you first edit the Oracle tnsnames.ora file to set up clustering connections. This is accomplished by opening the file in a text editor and assigning a Net Service name to the root node. Next, add an entry for each node using the same Net Service name plus a number (Figure 1). Figure 1. Setting up the cluster in the tnsnames.ora file for RAC testing. 9

13 Load Testing and Monitoring Oracle Real Application Clusters (RAC) ii. Creating a profile with RAC-distributed percentages Next, you create the Benchmark Factory profile to run the RAC load test. Profiles are sets of information that Benchmark Factory uses to communicate with the system under test. This information includes server name, IP address and user name. You create profiles using the Profile Creation Wizard. Benchmark Factory accomplishes RAC load balancing by setting up a RAC profile. In the profile settings, you set the desired number of nodes and percentages to distribute loads across the RAC system to simulate real-world conditions (Figure 2). RAC Node Percentages Figure 2. Setting up RAC node load testing percentages. iii. Running the RAC load test Benchmark Factory allows you to replay production workload and SQL scripts in different database environments, and places enormous stress on a database. By replaying production activity in a test database, Benchmark Factory enables you to identify potential performance problems before they occur in production. Such proactive testing ultimately reduces downtime, development costs and potential loss of revenue. Benchmark Factory allows you to import a variety of user loads, including: Industry-standard benchmarks to measure the performance of a system under test. Benchmark Factory provides the AS3AP, Scalable Hardware, TPC-B, TPC-C and TPC-D standard benchmarks Oracle trace files with bind variables Imports from various tools, such as Quest Performance Analysis SQL from the repository, for replay 10

14 White Paper Benchmark Factory comes with wizards that guide you step by step through the RAC testing process. The New Project Wizard, for example, allows you to create workloads to load test. After running the load tests, you can import an Oracle trace file workload by selecting the trace file, applying filtering options and then replaying the file. When Benchmark Factory imports the Oracle trace activity (Figure 3), it maintains concurrency between transactions and inter-transaction timings that simulate a production environment. Activity as it happened in production can be simulated in a test database using an unlimited number of concurrent users. Figure 3. Importing Oracle trace activity for replay in Benchmark Factory. When replaying production activity, you can simulate what-if scenarios. For example, what if your production workload increases 10 times? This allows you to predict potential performance problems when the workload increases. 11

15 Load Testing and Monitoring Oracle Real Application Clusters (RAC) iv. Viewing testing results and determining the right number of RAC nodes Analysis is the ultimate goal of a RAC load test. When running a test, you want results in both real time (Figure 4) and in a saved format. Benchmark Factory provides you with testing results that are easy to interpret and that allow you to attribute individual results to individual tasks and users. Whether you are viewing real-time or historical graphs, Benchmark Factory uses the same graphing tool, which allows you to customize graphs and tailor your load testing viewing requirements. Figure 4. Benchmark Factory real-time testing results. When deploying RAC environments, a major challenge is to determine the right number of RAC nodes to deploy for a given application or database, while minimizing the implementation and maintenance costs of RAC environments. Benchmark Factory for Databases can generate workload or replay real production activity against a different number of RAC nodes. Loads then can be balanced among the different nodes. For example, if you are testing two nodes, you can load test against one instance, while the other instance in the cluster is shut down. This in turn tunes the instance and operating system of the cluster being tested. After tuning the nodes, you can run load tests across all nodes in the cluster, specifying a goal or target. 12

16 White Paper Without Benchmark Factory for Databases, setting up the right number of RAC nodes becomes a trial-and-error process, lacking any real metrics. By contrast, with the level of detail provided by Benchmark Factory for Databases, companies can eliminate significant, unnecessary costs when implementing Oracle RAC environments. Users, for example, can accurately determine the specific number of nodes needed to meet scalability and performance requirements for their applications. Figure 5 shows RAC performance testing results in Benchmark Factory against different cluster nodes. Figure 5. Performance comparison in Benchmark Factory after workload replay in a RAC environment. 13

17 Load Testing and Monitoring Oracle Real Application Clusters (RAC) b. RAC Testing Using the Standard Load Balancing Criteria When using the standard load balancing criteria provided by Oracle to test each of the instances in the cluster, the first step is to ensure that all required parameters have been defined for the TNS connect descriptor used to connect to the database. i. Verify the TNS connect descriptor used to connect the database. Oracle provides several levels of load balancing of the cluster, such as client load balancing, server load balancing and, as of Oracle Database 10g Release 2, connect time load balancing. A discussion of the various types of load balancing is outside the scope of this paper. However, the minimum configuration required to enable load balancing of the cluster nodes is to define the TNSnames connect descriptor with the appropriate connection information, as shown in Figure 6. Figure 6. Verifying TNS connect descriptor. Once you have verified the TNS connect descriptor using Benchmark Factory for Databases, the next step is to define a load testing profile based on the type of test and the criteria to be tested. 14

18 White Paper ii. Creating a profile Profiles are created using the Profile Creation Wizard, as shown in Figure 7. This information includes server name, IP address and user name. Figure 7. Benchmark Factory Profile Creation Wizard. iii. Running the RAC load test Once the profile definition is complete, the next step is to perform the actual tests. Benchmark Factory provides several types of testing options. You can choose to use either the industry-standard benchmark tests or methods to replay production workload and SQL scripts from different database environments. The goal of testing with either method outlined above is to help you monitor a RAC database for any contentions, and appropriately diagnose areas that can be tuned to improve overall performance of the cluster. 3. Monitoring RAC with Quest Spotlight on RAC Spotlight on RAC from Quest Software provides options for problem identification and diagnosis. Once a RAC system has been set up with the correct number of nodes and load tests, the next phase of database performance monitoring begins. DBAs must know: How instances in the cluster perform How the database performs as a whole Whether the database is experiencing cluster-related contention 15

19 Load Testing and Monitoring Oracle Real Application Clusters (RAC) RAC is a shared disk clustering solution. Each node in RAC has equal access to all of the database data through a shared disk subsystem. Data is neither partitioned to specific nodes, nor replicated across nodes. A high-speed network interconnect allows each node to keep its in-memory view of data consistent. An Oracle RAC configuration (Figure 8) has the following architectural characteristics: Many instances of Oracle run across several nodes. Many instances share a single physical copy of a shared Oracle database. All instances have common data and control files. Each instance has individual redo logs and undo segments. Each of the instances simultaneously executes transactions against the same database. Public Interface Cluster Interconnect SSKY1 SSKY2 SSKY3 SSKY4 ORADB1 ORADB2 ORADB3 ORADB4 Oracle Database Figure 8. Oracle RAC configuration 1. Spotlight on RAC presents a full diagnostic capability (Figure 9) for an Oracle RAC environment. It takes RAC monitoring to a whole new level by issuing queries on each instance in the cluster, and then performing the appropriate calculations to provide RAC-wide diagnostic information. 1 Source: Oracle 10g RAC, Grid, Services & Clustering by Murali Vallath (used with publisher s permission) 16

20 White Paper Figure 9. Spotlight on RAC diagnostic information. Spotlight on RAC evaluates seven RAC-specific alarms (Figure 10), providing diagnostic information explicitly related to RAC and raising a diagnostic alarm if: The database load is not evenly distributed across instances within the cluster. The overhead of maintaining consistency across the cluster is excessive (e.g., the database is experiencing degradation because it is part of a cluster). There is high latency across the interconnect that links instances in the cluster. There is significant inter-instance communication (which might indicate the need for data partitioning). Blocks are becoming corrupt or lost while being transferred between the instances in the cluster. The shared disk subsystem is experiencing contention. 17

21 Load Testing and Monitoring Oracle Real Application Clusters (RAC) Figure 10. Spotlight on RAC alarms. Spotlight on RAC adds a whole new dimension to the everyday supervision of RAC. By gathering details from each of the instances within the cluster, it delivers relevant diagnostic information at both the instance and cluster-wide levels. Spotlight on RAC provides: 18 Descriptive representation of RAC physical architecture Spotlight on RAC's home page displays an informative view of the key physical components of a cluster: the individual instances, interconnect and I/O subsystem, and the data flows between them. Drilldown into Spotlight on Oracle If a particular instance in the cluster displays problems (via color highlighting on the Spotlight on RAC home page), you can drill down into Spotlight on Oracle, which underlies Spotlight on RAC, with a simple click. Aggregated Oracle alarms Oracle problems such as latch contention could occur on any particular instance in the cluster. Whether this is cause for concern depends on whether the problem extends to most or all of the instances in the cluster. Spotlight on RAC aggregates all the data from the instances and informs you whether the cluster as a whole has a problem. RAC-specific alarms Various problems are specific to RAC installations: unbalanced load, cluster latency and overhead, corrupt and lost blocks, and cache misses. Spotlight on RAC evaluates and raises RAC-specific alarms that impact the overall health and performance of the cluster. Support for Automated Storage Management (ASM) Spotlight on RAC monitors the operation of ASM, providing associated diagnostic information.

22 White Paper THE UNIQUE CAPABILITIES OF BENCHMARK FACTORY FOR DATABASES AND SPOTLIGHT ON RAC Benchmark Factory offers unique RAC load testing capabilities for database professionals who need to guarantee that their RAC applications will scale. Unlike other load testing tools, Benchmark Factory is an out-of-the-box, database load testing system that requires no programming or extensive setup, providing complete RAC testing capabilities that: Allows testers to use trace files from actual production databases to test RAC systems Provides industry-standard benchmarks that allow realistic testing of RAC nodes Provides both real-time and stored testing results to analyze RAC nodes Allows testers to play what-if scenarios to determine what effect actual database workloads have on a RAC system Allows testers to run load tests against individual nodes or several nodes at once Provides a graphical breakdown of individual RAC nodes to determine actual performance based on real-world activity Spotlight on RAC is a powerful diagnostic and resolution tool for Oracle databases running under RAC architecture. Its unique user interface provides you with an intuitive, visual representation of the activity on your database and quickly becomes an indispensable tool to keep your RAC system operating at peak performance. Spotlight on RAC allows IT professionals to: View combined information from all instances to receive a spotlight view of the cluster as a whole View a graphical representation of the cluster architecture Monitor global health and activity View instance-level health and activity Track RAC-specific problems that include: Interconnect latency Unbalanced cluster Excessive cluster overhead 19

23 Load Testing and Monitoring Oracle Real Application Clusters (RAC) CONCLUSION Oracle Real Application Clusters offers a means of providing scalability and availability of the Oracle database environment. By load testing a RAC system, DBAs can measure performance failover and availability of component clusters, and get a clear view of database efficiency. Moreover, they can gain insight into system performance and better ensure continuous customer satisfaction. An effective database must scale with the number of users and respond reliably; it also must be monitored to ensure a smooth data flow from server to user. Quest Benchmark Factory for Databases and Spotlight for RAC simplify the RAC load testing and monitoring process by providing ease of use, accuracy and the data points necessary to ensure your RAC system performance not only meets, but exceeds customer and end-user expectations. 20

24 White Paper ABOUT THE AUTHOR Claudia Fernandez is the senior product manager of database performance and scalability products at Quest Software. Claudia has contributed to the strategic direction of database tuning products for multiple RDBMS since early She has presented at several technical conferences on RDBMS and application performance tuning topics. Claudia holds an M.S. in computer science and has 10 years of industry experience working with different databases, such as SQL Server, Sybase ASE, Oracle, DB2 UDB and other associated technologies. Bernard Farrell is a technical writer at Quest Software, Inc., writing technical documentation on Benchmark Factory for Databases. He has 15 years of technical writing experience, with five years of experience documenting database applications. 21

25 Load Testing and Monitoring Oracle Real Application Clusters (RAC) ABOUT QUEST SOFTWARE, INC. Quest Software, Inc. delivers innovative products that help organizations get more performance and productivity from their applications, databases and Windows infrastructure. Through a deep expertise in IT operations and a continued focus on what works best, Quest helps more than 18,000 customers worldwide meet higher expectations for enterprise IT. Quest Software can be found in offices around the globe and at Contacting Quest Software Phone: Mail: Web site (United States and Canada) info@quest.com Quest Software, Inc. World Headquarters 5 Polaris Way Aliso Viejo, CA USA Please refer to our Web site for regional and international office information. Contacting Quest Support Quest Support is available to customers who have a trial version of a Quest product or who have purchased a commercial version and have a valid maintenance contract. Quest Support provides around the clock coverage with SupportLink, our web self-service. Visit SupportLink at From SupportLink, you can do the following: Quickly find thousands of solutions (Knowledgebase articles/documents). Download patches and upgrades. Seek help from a Support engineer. Log and update your case, and check its status. View the Global Support Guide for a detailed explanation of support programs, online services, contact information, and policy and procedures. The guide is available at: Support Guide.pdf 22