Consolidating DSS Workloads on Dell PowerEdge 11G Servers Using Oracle 11g Database Replay

Transcription

1 Consolidating DSS Workloads on Dell PowerEdge 11G Servers Using Oracle 11g Database Replay A Dell Technical White Paper Database Solutions Engineering By Zafar Mahmood Dell Product Group April 2009

2 THIS WHITE PAPER IS FOR INFORMATIONAL PURPOSES ONLY, AND MAY CONTAIN TYPOGRAPHICAL ERRORS AND TECHNICAL INACCURACIES. THE CONTENT IS PROVIDED AS IS, WITHOUT EXPRESS OR IMPLIED WARRANTIES OF ANY KIND. For more information, contact Dell. Intel and Xeon are registered trademarks of Intel Corporation in the U.S. and other countries; Oracle is a registered trademark of Oracle Corporation. Quest Software and Benchmark Factory are registered trademarks of Quest Software, Inc. Other trademarks and trade names may be used in this document to refer to either the entities claiming the marks and names or their products. Dell disclaims proprietary interest in the marks and names of others. 2

3 EXECUTIVE SUMMARY Change management is one of the most important factors to consider for smoothly running and maintaining an enterprise data center. This is especially true of mission critical business applications that rely on a backend database system any type of permutation into the production environment is risky if not governed by carefully defined rules and procedures. This is why most data centers follow a very conservative approach when implementing, deploying, and utilizing recently introduced software features. They follow the same approach when server hardware and storage system vendors introduce the latest feature rich technologies claiming to offer better performance. Dell s portfolio of enterprise products is evolving to incorporate better performing, more energy efficient, and more highly available features. With the latest introduction of the Dell 11G server product line, customers have an opportunity to improve their total cost of ownership by consolidating dispersed legacy environments on this product line. However, many enterprise customers are reluctant to upgrade to new solution stacks due to the following factors: Lack of appropriate diagnostic and performance testing tools to validate the performance and stability of new hardware and software solution stacks Unavailability of realistic data on the test environment Unavailability of realistic workloads and user patterns that mimic the production environment Because of these limitations, enterprise customers may hesitate to adopt the latest advancements in server, storage, and cluster interconnect technologies. These issues are also proving a hindrance to utilizing alternatives that are more energy efficient than old legacy systems in other words, consolidating energy hungry legacy systems into fewer, more powerful and energy efficient systems. With the introduction of the 11G server platform, Dell strives to simplify IT infrastructure by consolidating legacy production environments to reduce data center complexity while still meeting customers needs. The tools and procedures described in this white paper can help administrators test, compare, validate, and implement the latest hardware and database solution bundles offered by Dell. Dell established these procedures and guidelines based on lab experiments and database workload simulations performed by the Dell Database Solutions Engineering team using the Oracle 11g Database Replay component. Using the tools and procedures described in this document, customers may not only select the appropriate database solution hardware and software stack, but also optimize the solution to help optimize total cost of ownership according to the database workloads they intend to run. Although Database Replay is intended to help alleviate the pains of upgrades and migrations from older releases to 11g and later, this white paper discusses server consolidation as the main use case. The intended audience of this white paper includes database administrators, IT managers, and system consultants. 3

4 CONTENTS EXECUTIVE SUMMARY... 3 INTRODUCTION... 5 OVERVIEW OF DATABASE REPLAY... 6 TEST METHODOLOGY... 7 TEST CONFIGURATION RESULTS CONSOLIDATION FACTOR SUMMARY REFERENCES

5 INTRODUCTION Before implementing system changes such as hardware and software upgrades, organizations should perform extensive testing to validate the impact on stability and performance. Even with testing, however, new solution stacks often experience unexpected behavior in production because tests do not use realistic workloads. Because production system usage patterns and user behaviors are absent in a test environment, it is difficult to simulate a realistic workload during testing to thoroughly validate system changes. An Oracle 11g feature called Database Replay helps mitigate this challenge. Database Replay is designed to enable realistic testing of system changes by essentially recreating a sample of the production workload environment on a test system. Database Replay captures the workload on a production system and recreates it on a test system with the exact timing, concurrency, and transaction characteristics of the original workload. Using this information, you can assess the impact of the solution change, including undesired results, new contention points, and plan regressions. The Database Replay feature also provides the tools required to analyze and report on the replayed workload using either the built in PLSQL API or GUI tools such as Oracle Enterprise Manager Database Control and Grid Control. Since the capture and replay workload environment mimics the data and usage patterns of the production environment, it helps eliminate the need to develop simulation workloads and scripts, resulting in significant cost reduction and time savings. Using Database Replay, realistic testing of complex applications that previously took months using load simulation tools can now be completed in days. This enables database and system administrators to rapidly test changes and adopt new technologies with higher confidence and lower risk. Database Replay performs external client workload capture at the database level, without the need to configure application servers and with negligible performance overhead. Database Replay can be used to test most significant system changes, including but not limited to: Database and operating system upgrades Configuration changes, such as converting a database from a single instance to an Oracle Real Application Clusters (RAC) environment or vice versa Storage, network, and interconnect changes Operating system and hardware migrations Server and storage consolidation This white paper concentrates on a server consolidation usage model for Database Replay. Server consolidation can be defined as maximizing the efficiency of computer server resources and minimizing associated power costs to reduce the total number of servers that an organization requires. It essentially solves a fundamental problem called server sprawl in which multiple, under utilized servers take up more space and consume more power resources than the workload requirement indicates. For example, consider a two node Oracle RAC database hosted on two eighth generation (8G) PowerEdge 2850 dual socket, single core or dual core servers running Oracle 10g Release 2. Dell recently announced the availability of its 11G server product line equipped with a chipset that is designed to support the Intel Nehalem EP processor family, QuickPath Interconnect, DDR3 memory technology, and PCI Express Generation 2. The natural replacement of eighth generation 2U Dell servers is 2U Dell R710 11G servers that support dual socket, quad core processors. The R710 also supports two different types of energy efficient CPUs, and it is designed with a highly efficient overall architecture. The goal of this study is to replace a multi node Oracle RAC cluster with a cluster consisting of fewer PowerEdge 11G nodes, and still process the workload faster with less power consumption and lower Oracle RAC licensing fees. 5

6 This type of consolidation is not only possible, but can also result in an exceptional performance boost. The lab results described in this white paper overwhelmingly favor this type of consolidation. OVERVIEW OF DATABASE REPLAY As depicted in Figure 1, the Oracle 11g database replay framework consists of four processes: Workload Capture During peak production hours, the production system workload is captured or recorded on the file system as binary files. All client requests are recorded in the exact order and frequency in which they are received. These requests may include SQL queries and DML operation. The only exceptions are background database and Oracle Scheduler jobs. The captured files are platform independent, so they can be transferred and replayed on any platform running Oracle 11g R1 or higher. Oracle 10g R2 Patchset 4 ( ) adds the capture capability to Oracle 10g R2. Workload Preprocessing After the workload is captured into the binary files, the files must be preprocessed, including creating all the necessary metadata needed to replay the workload on the test system. Dell recommends that you preprocess the workload on the test system where it will be replayed. Once the workload is preprocessed, you can repeatedly replay it on any test system running the same database version where it was preprocessed. Workload Replay Replaying the captured and preprocessed workload requires starting and running the required number of database replay clients preferably on a system other than the database server. During the workload replay phase, Oracle performs the actions recorded during the workload capture phase on the test system by recreating all captured external client requests with the same timing, concurrency, and transaction dependencies of the production system. A calibration tool is provided to help determine the number of replay clients needed for a particular workload. Because the entire workload is replayed, including DML and SQL queries, the data in the replay system should be logically similar to the data in the capture system to minimize data divergence and enable reliable replay analysis. Reporting and Analysis After the workload replays, Oracle provides in depth reporting of both workload capture and replay for you to analyze. You can run all of these processes using either the built in PL/SQL packages or Oracle Enterprise Manager (DB Control or Grid Control). 6

7 Figure 1: Database Replay Architecture TEST METHODOLOGY Dell s solution engineers used Quest Software Benchmark Factory Scalable Hardware CPU intensive workload to be captured on the legacy system and then replay in a test environment running PowerEdge 11G servers with three different CPU configurations. The CPU intensive workload provided by the Benchmark Factory schema consists of parts and products tables that are populated with rows according to the scale factor defined during table creation. We configured the legacy database with a scalability factor of 10,000, which created 400 Million and 100 Million rows in the parts and products tables, respectively. The total database size that resulted with this scale factor was around 75GB. Once populated, we simulated 20 users randomly running complex queries against the legacy database. The test methodology we used is as follows: 7 1. To simulate the legacy production environment, we selected a two node Oracle 10g R2 RAC cluster running on two PowerEdge 2850 single core, dual socket 3.4 GHz CPU machines connected to a CX4 960 that had two 100GB and one 20GB LUNs for voting and Oracle Cluster Registry (OCR). 2. We applied the Oracle 10g R2 patchset 4 ( ) to the legacy server simulated production environment to enable database capture support. 3. We loaded test data with a scalability factor of 10,000 into the legacy server simulated production environment.

8 4. After data population, we used Oracle Data Pump to export data at the schema level for later population in the test environment, which ran on PowerEdge 11G servers with the following script: expdp SCHEMAS=quest CONTENT=all directory=export; 5. In the patchset, we used a new initialization parameter to enable workload capturing. This parameter is disabled by default, as follows: SQL> show parameter pre_11g_enable_capture NAME TYPE VALUE pre_11g_enable_capture boolean FALSE To enable capture on Oracle 10g R2 patchset 4 databases, we used the following Oracle script, which sets the initialization parameter to enable workload capture capability: 6. In order to store the captured workload files, we created a directory object on the legacy system as follows: SQL> create directory capture as '/orafiles/capture/'; 7. We created filters to preclude capturing user sessions that produce activity that can be ignored during the capture period. You should create either an inclusion filter that defines the user sessions that must be captured or an exclusion filter that defines which user sessions must be excluded from the capture process. One example of an exclusion filter excludes Enterprise Manager Database Control or Grid Control activity using either the built in DBMS_WORKLOAD_CAPTURE.ADD_FILTER package or Enterprise Manager. 8. At this point, the simulated legacy production environment was ready to capture the workload. We generated a workload that was configured to run for one hour, and we enabled the workload capture to capture system activity for the entire one hour period, as shown in Figure Once the capturing process completes, Oracle Enterprise Manager Grid Control displays the capture statistics as shown in Figure We took an Automatic Workload Repository (AWR) snapshot of database activity in the legacy production environment for inclusion in the capture files we ll use this snapshot later for comparison with the test environment at time of replay. You can also accomplish this by using the PLSQL package DBMS_WORKLOAD_REPOSITORY.create_snapshot or Enterprise Manager. 11. Using the Oracle Data Pump Import utility, we populated the Oracle 11g environment running on the test environment. We used a PowerEdge R710 server to simulate our test environment. impdp system/oracle@zafar_1 SCHEMAS=quest CONTENT=ALL directory=dp_import 12. We moved the workload capture files from the capture directory to the replay system. You can use either SCP or FTP, and create a directory object on the replay test system with the following script: create or replace directory "replay" as '/opt/oracle/replay'; 8

9 13. Next, we preprocessed the captured workload in the test environment. Again, you can use either use the DBMS_WORKLOAD_REPLAY.PROCESS_CAPTURE package or Enterprise Manager Database Control (EMDB control) for this step. Figure 2: Workload Capture Setup Now, we were ready to replay the workload. The replay setup requires you to remap the database connections from the captured workload to the new test environment. You must consider any changes to the host names, number of nodes, and service name. If the legacy environment consists of a single node and the test environment is a Real Application Cluster (RAC), you may need to create database service and configure it to use Connection Load Balancing (CLB) and Load Balancing Advisor (LBA) for the shortest connection times. Also, you should add a service description to the Oracle client naming file using the following script: SQL> begin DBMS_SERVICE.MODIFY_SERVICE ( service_name => 'zafarsvc',goal => DBMS_SERVICE.GOAL_SERVICE_TIME,clb_goal => DBMS_SERVICE.CLB_GOAL_SHORT,aq_ha_notifications => TRUE ); END; /

10 If moving from RAC to a single node environment, as in this test, you can use a single connect descriptor for all RAC connections. You can either use EMDB control or DBMS_WORKLOAD_REPLAY. remap_connection to accomplish this task. Figure 3: Workload Capture Report 15. After remapping the connections, we prepared the replay parameter in terms of synchronization, think time, and connect time to customize the behavior of the replay. The synchronization parameter is important, since the commit order of all transactions is preserved when set to true. In our testing, we set the parameter to false, since we used a read only workload and allowed the transactions to complete without waiting for the order to take advantage of the enhanced processing power of the PowerEdge 11G system. 16. We started the database workload replay client(s) (WRC) on a different system than the one we were testing. Before starting the WRC, you can use the calibration option to determine how many WRC clients are required to replay the workload. 10 [oracle@replay1 ~]$ wrc system/oracle@replay mode=calibrate replaydir=/opt/oracle/replay/

11 The calibration mode suggested using one WRC client for our workload: ~]$ wrc replaydir=/opt/oracle/replay/ Figure 4 shows the WRC client ready and waiting for replay to start: Figure 4: Workload Replay Client Waiting for Replay to Start 17. Finally, once the replay completed, we viewed replay progress and reports. You can use either DBMS_WORKLOAD_REPLAY or the Enterprise Manager GUI for this step, as shown in Figure 5. The replay report shows the amount of time it took to replay the workload in the test environment as compared to the legacy production environment. In Figure 5, we see that the test environment running Oracle 11g Release 1 with one Intel Xeon X5570 processor took only 17 minutes and 59 seconds to run the workload that the legacy two node cluster running on PowerEdge eighth generation 2850 servers took one hour to complete. 11

12 Figure 5: Database Replay report TEST CONFIGURATION Table 1 describes the complete software and hardware stack that we used throughout testing on both our simulated legacy production environment and the test environment running on the latest Dell PowerEdge 11G servers. Table 1: Oracle 11g Database Replay Test Configuration Component Legacy Production Environment Dell PowerEdge 11G Test Environment Systems Two PowerEdge U servers One PowerEdge R710 2U server Processors Two Intel Xeon CPU 3.40GHz single core per Test1: One Intel Xeon X5570 node Cache: L2=1M per CPU 2.93GHz quad core CPU Cache: L2=4x256K L3=8M 12 Test2: One Intel Xeon L GHz quad core CPU

13 Cache: L2=4x256K L3=8M Test3: One Intel Xeon L GHz quad core CPU Cache: L2=4x256K L3=4M Memory 8GB DDR2 per node 8GB DDR3 Internal disks Two 73GB 3.5 SCSI R1 Two 73GB 2.5 SAS R1 Network Two Intel 82544EI Gigabit Ethernet Four Broadcom NetXtreme II BCM5709 Gigabit Ethernet External storage Dell/EMC CX4 960 with fourteen 146GB Fibre Channel disks Dell/EMC CX4 960 with fourteen 146GB Fibre Channel disks HBA Two QLE2460 per node Two QLE 2460 OS Enterprise Linux 4.6 Enterprise Linux 5.2 Oracle software Oracle 10g R File System: ASM Diskgroups: DATA, FLASH sga_target = 1600M pga_target = 800M Oracle 11g R File System: ASM Diskgroups: DATA, FLASH memory_target = 2400M Workload Quest Benchmark Factory CPUintensive workload Scale factor: 10,000 User connections: 20 Replayed workload RESULTS NOTE: The results we have provided are intended only for the purpose of comparison between two environments consisting of specific configurations in a lab environment. The results do not portray the maximum capabilities of any system, database software, or storage. The following test results address questions regarding Oracle 10g capture and replay in Oracle 11g environments in order to provide server consolidation guidelines on PowerEdge 11G servers. The goals of the test were to determine: 13 How system and database administrators can take advantage of the Oracle Database Replay feature to help ease the pains of change management, database migration, performance assessments, and server consolidation The watt hour consumption comparison between test and legacy production environments How Oracle workloads can take advantage of the energy efficient architecture of PowerEdge 11G servers An ideal energy efficient CPU option on which to run read only CPU intensive Oracle workloads for Dell 11G servers The consolidation factor resulting from migrating to PowerEdge 11G servers from PowerEdge 8G singlecore servers running read only CPU intensive workloads

14 After completing the database replay on all three CPU configurations, we analyzed our results to determine which R710 configuration provides the best results in terms of watt hours used. We analyzed the power consumption chart for all configurations throughout the capture and replay periods, as shown in Figure 6. Power Consumption (Watts) W a t t s Time in Minutes 60 Watts 2850 node1 Watts 2850 node2 Total Watts 2x2850 Watts R710 (X5570) Watts R710 (L5520) Watts R710 (L5506) Figure 6: Power consumption during workload capture and replay As shown in Figure 6, our two node legacy Oracle RAC configuration consumed an average of 685 watts while running a 60 minute workload. On the other hand, our R710 populated with varying CPU configurations X5570, L5520, and L5506 consumed 195, 161, and 151 watts on average, respectively. To calculate the total watt hours consumed by each configuration, we combined the results provided by the database replay workload report showing the capture versus the replay time (see Figure 7). 14

15 Consolidating Workloads on Dell PowerEdge 11G Servers Using Oracle 11g Database Replay Time comparison between capture and replay systems (Sec) T i m e ( S e c ) x2850 Capture nodes R710 (X5570) R710 (L5520) R710 (L5506) Time (Sec) Figure 7: Comparison of Capture and Replay Systems Figure 7 shows that an R710 populated with a single X5570 quad core CPU takes only 17:58 (mm:ss) to replay the same workload that was captured in one hour on the two node 2850 RAC. The energy efficient variations of the Intel Xeon CPUs (L5520 and L5506) also performed exceptionally well, and replayed the whole workload in only 21:36 and 26:01, respectively. To validate our results, we ran the CPU intensive workload on both the simulated legacy production environment and the test environment without using Database Replay and measured the transactions per second. The results, shown in Table 2, validate our database replay report, which also exhibited a 234% performance gain in terms of workload replay completion time. Table 2: Results Validation Concurrent User Load Simulated Legacy Production Environment (2x2850) Dell PE 11G R710 (X5570) Server Performance Delta DB Replay (seconds) Results seconds 1078 seconds 11G: 234% faster BMF Results (TPS) G: 224% more TPS To calculate the total watt hours for each configuration, we multiplied the average watt consumption with the workload run time and plotted the results, as shown in Figure 8. Figure 8 also shows how watt hour consumption relates to average CPU utilization. 15

16 Consolidating Workloads on Dell PowerEdge 11G Servers Using Oracle 11g Database Replay Workload Replay:Watt Hours and CPU Consumption Most Efficient Configuration Watt Hours consumed to run the workload Average CPU Utilization Capture node Capture node2 2x2850 Capture nodes R710 (X5570) R710 (L5520) R710 (L5506) Figure 8: Finding the most efficient CPU Configuration Using Watt-Hour Consumption Although the R710 configuration populated with the Xeon Low Voltage L5520 processor took 1296 seconds to replay the workload as compared to only 1078 seconds for the X5570 processor, both consumed approximately the same watt hours during replay as shown in Table 3. Thus, we can conclude that both X5570 and L5520 processorss are good candidates to run this CPU intensive workload if the criteria is energy efficiency. If maximizing the query response time is the objective, then the X5570 processorr configurationn should be our choice. Also, using the watt hour consumption data, we can conclude that the L5506 processor configuration is least optimal in terms of watt hour consumption for this type of workload. Both X5570 and L5520 processor configurations also exhibited only 80.95% and 85.54% average CPU consumption as compared to almost 99% by our capture systems consisting of a two node Oracle RAC as shown in Figure 8.. Table 3: Watt Hour Consumption Comparison During Workload Capture and Replay Configuration Ave Watts Total Time (seconds) Ave Watt Hours 2850 capture node capture node Two 2850 capture nodes R710 (X5570)

17 R710 (L5520) R710 (L5506) A primary feature of 11G servers is power efficient design that uses energy efficient components such as: DDR3 memory (lower voltage than DDR2) Speed modulated fans with an increased number of fan zones and configuration dependent fan speeds compared to the prior generation BIOS power/performance options Ability to power down or throttle memory Ability to disable a CPU core Ability to turn off embedded NICs or PCI express lanes when unused Option to run PCI express at Gen1 speeds instead of Gen2 when needed These energy efficient features result in lower power consumption in an idle state as compared to our legacy production environment (see Figure 9). W a t t s Idle State Power Consumption Sample Number Watts 2850 node1 Watts 2850 node2 Total Watts 2x2850 Watts R710 (X5570) Watts R710 (L5520) Watts R710 (L5506) Figure 9: Idle State Power Consumption Consolidation Factor Based on test results, we can conclude that an R710 server populated with a single quad core Intel Nehalem based CPU (X5570) was able to process the workload of a two node Oracle RAC running on 8G servers populated with 17

18 dual single core processors, up to 224% faster with up to 81% average CPU utilization. We can safely extrapolate that if we had populated our 11G R710 server with dual quad core Nehalem based processors, we could have handled the workload up to 200% faster than four 8G dual socket single core servers, with significantly less power consumption. The power savings that result from consolidating on PowerEdge 11G servers alone could offset the cost of the new servers very quickly. The consolidation factor of 4:1 8G servers versus one fully populated 11G server could be doubled to 8:1 if the CPU utilization on the Oracle 10g RAC nodes consisting of 8G single core processors were 50% which is usually the Service Level Agreement (SLA) maintained by enterprise data centers on backend database servers to provide a cushion for unexpected workload spikes. On both capture and replay environments, the IOWAIT was negligible since the workload was processor bound. The legacy environment exhibited CPU bottleneck which was mitigated by consolidating the workload on PowerEdge 11G server. Also, both capture and replay environments showed no bottleneck in terms of IO or memory as shown in Table 4: Table 4: System Status Report (VMSTAT Averages) System Average CPU User System IOWAIT Free Memory(G) Utilization 2850 Capture node Capture node Two 2850 capture nodes R710 (X5570) R710 (L5520) R710 (L5506) SUMMARY Any server consolidation project must be preceded by a well planned effort to predict the energy consumption and performance capacity of the new platform as compared to the legacy environment. In this study, we showed how you can use the Oracle Database Replay feature as a reliable tool to help in server consolidation projects. We also showed that PowerEdge 11G servers equipped with Xeon 5500 Series chipsets for I/O and processor interfacing provide an ideal platform to consolidate legacy database environments. The R710 chipset is designed to support Intel s Nehalem EP processor family, QuickPath Interconnect, DDR3 memory technology, and PCI Express Generation 2. This study also indicates that 11G servers offer large performance gains when compared to older generations of servers with front side bus architectures. Customers running Oracle 9i or 10g RAC environments on legacy servers and storage can follow the guidelines and procedures outlined in this white paper to consolidate power hungry RAC nodes into fewer, faster, more energyefficient nodes. The resulting legacy RAC node consolidation can also drive down Oracle licensing costs, resulting in savings that you can use to implement disaster recovery sites and additional RAC test bed sites for application development and testing. The reduced number of nodes does not compromise performance when paired with PowerEdge 11G servers. The result is less cluster overhead, simplified management, and positive movement toward an objective of simplifying IT and reducing complexity in data centers. REFERENCES 18 Oracle Database Performance Tuning Guide 10g Release 2 (10.2) Part Number B Oracle Database Real Application Testing User's Guide 11g Release 1 (11.1) Part Number E