Quality Center Benchmark March 2009

Transcription

1 Quality Center Benchmark March 2009 Author Date Comment Version David Elimelech, Software Performance Engineer 30/04/ Audit Trail: Reviewer Date Comment Version Eitan Tepper, PCoE-AD 30/04/ Team Manager Technical writer 15/05/09

2 Legal Notices Warranty Hewlett-Packard makes no warranty of any kind with regard to this manual, including, but not limited to, the implied warranties of merchantability and fitness for a particular purpose. Hewlett- Packard shall not be held liable for errors contained herein or direct, indirect, special, incidental or consequential damages in connection with the furnishing, performance, or use of this material. Restricted Rights Legend Use, duplication or disclosure by the U.S. Government is subject to restrictions as set forth in subparagraph (c)(1)(ii) of the Rights in Technical Data and Computer Software clause in DFARS Hewlett-Packard Company United States of America Rights for non-dod U.S. Government Departments and Agencies are as set forth in FAR (c)(1,2). Copyright Notices Mercury Interactive Corporation, All rights reserved Trademark Notices Microsoft and Windows are U.S. registered trademarks of Microsoft Corporation. Microsoft SQL Server is a trademark of the Microsoft Corporation. Oracle is a registered US trademark of Oracle Corporation, Redwood City, California. UNIX is a registered trademark of The Open Group. Red Hat Linux is a trademark of Red Hat, Inc. Solaris is a registered trademark of Sun Microsystems, Inc WebLogic is a trademark of Oracle Corporation WebSphere is a trademark of IBM Corporation. Jboss is a trademark of JBoss Inc.

3 Table of Contents 1. Introduction Purpose and Scope Terms and Definitions Executive Summary Recommended Environments Sizing and Scalability Configuration Lab Topology Labs Specifications Usage Profile Overview Data Profile Data Profile Overview Data Profile Structure Data Profile Sizing Recommended Environment Results Recommended Environment Configuration Transactions response time distribution System Resources Summary Table Sizing / Scalability Sizing/Scalability Configuration Sizing/Scalability Results System Resources Summary Table JVM Tuning General Tuning Sun JVM on JBOSS Additional JVMs Weblogic on Linux Red Hat WebLogic on Sun Solaris JBOSS on HP-UX Appendix I. Tested Profile II. Data Profile by Entities III. Data Profile by Table Size... 22

4 - HP Restricted Introduction 1.1 Purpose and Scope This document provides information gathered during the benchmark performed on QC Atlantis (v10). This standardized benchmark is used for both internal release testing and environment validation at customer deployment efforts in the field. It is divided into the following sections: Benchmark design Provides a detailed description of the benchmark profile. Recommended environment results Provides contextual data and results regarding the environment in which each benchmark scenario was performed. Test results for the QC benchmark are provided for reference purposes as they are executed within QC performance lab. Sizing and scalability results - Determines the hardware needed to hold the QC nodes according to the amount of active users with the load, as defined in section 3.3. This benchmark was run on multiple environment configurations chosen for their prevalence in today s QC customer install base. The scenario results are not meant to serve as basis of extrapolating suitability of one configuration over another because of variables such as hardware, operating system, application server, DB type and DB system parameters. 1.2 Terms and Definitions Term Vuser Transaction Transaction Response Time Active users Business Process Definition Virtual user. Simulation of real user performing the actions on the product through LoadRunner scripts. A measurement of one or more user actions that has a tangible business value to the end-user. The time measured by LoadRunner generator as the response time of the transaction. This is defined by LoadRunner as the time taken for all web requests belonging to the transaction to return the responses. It is also called Transaction Latency. Connected users that execute activity at this moment, can be concurrent (but not necessarily so). A business process represents a collection of discrete application transactions that are executed together in sequence. Each LoadRunner script created for the benchmark represents a business process.

5 2 Executive Summary 2.1 Recommended Environments Table QC V10.0 deployment has several recommended environments. Four recommended environments that were tested, achieved a satisfying average transaction response time, and good resources management over executions and load profile changes (CPU, memory and I/O resources). Recommended environments were tested with full load of 500 concurrent users. For complete results and details, see section 4. Env Environment Configuration 500 Users Full Load For Recommended Env App: JBOSS on Win2003 DB: MSSQL 2005 App: WebLogic on Red Hat Linux DB: Oracle 10 App: JBOSS on HP-UX DB: Oracle 10 App: WebLogic on Sun Solaris DB: Oracle Sizing and Scalability The environments described in Table 1.4.2, took part of the QC scalability and sizing benchmark. Three QC deployment environments were tested, including small, medium and large. The DB size and DB server strength were constant throughout the process. The changed parameters were the amount of users, hits per second, and the amount of QC nodes. Table Env Environment Configuration Small Env. Nodes:1 Users:500 Medium Env. Nodes:1 Users:1000 Large Env. Nodes:2 Users: App:JBOSS on Win2003 DB: MSSQL 2005 App: WebLogic on Red Hat Linux DB: Oracle 10 App:JBOSS on HP-UX DB: Oracle 10

6 All three environments including the three sizing scenarios, achieved a satisfying average transaction response time, and good resources management over executions and load profile changes (CPU, memory and I/O resources). It can be affirmed that there are no significant degradations in response time and resources between the different types of environments. QC can be both scaled up to maximum amount of ~1000 users, and scaled out to run on multiple QC servers with minimum impact to performance. For complete results and details, see section 5.

7 3 Configuration 3.1 Lab Topology The diagram below represents the lab topology of a large environment. Small and medium environments have the same diagram except for the fact that they have only 1 application server node and no load balancer.

8 3.2 Labs Specifications Purpose Type CPU RAM HP UX Application Node 1 & 2 Linux Application Node 1 & 2 Windows Application Node 1 SUN Application Node 1 File Server DB For Windows Only DB ia 64 HP server rx4640 HP ProLiant BL460c G1 HP Proliant BL460 G1 Solaris SunFire V480 ProLiant BL460c G1 HP ProLiant DL380 G3 HP ProLiant BL460c G1 4 * 1.6 GHz (Intel Itanium) 4 * 1.86 GHz (Intel Xeon) 4 * 3.0 GHz (Intel Xeon) 2 X 900 MHz (Ultra SPARC III) 4 * 2.66 GHz (Intel Xeon) 4 * 3.1 GHz (Intel Xeon) 8 * 3.16 GHz (Intel Xeon) 8 GB 8 GB 4 GB 4 GB 2 GB Operating System HP-UX v3 64-bit Linux Red Hat bit Win 2003 x64 SP2 Sun Solaris bit Linux Red Hat bit 4 GB Win 2003 SP2 32 GB Linux Red Hat bit 3dr Party Software JBOSS WebLogic 9.2 Apache 2.2 JBOSS WebLogic 9.2 Apache 2.2 MSSQL 2005 SP2 Oracle Network: Bandwidth between all lab components was 1 Gbps. All components were in same VLAN. (Latency < 1msec) DB Storage used for MSSQL: Parallel SCSI using RAID 5 fault tolerance. Independent Array accelerator enabled. DB Storage used for Oracle: Type: HP StorageWorks Enterprise Virtual Array (EVA) 8100 Fault Tolerance: Data - vraid5 REDO - vraid1 Cache: Control Cache: 4GB Data Cache: 4GB Disk Group consists of 80 disks, 450GB 15k FC each.

9 3.3 Usage Profile Overview As the usage profile may vary between QC customers, the benchmark test profile aims to give a full picture of the product s performance by covering the main functionalities of each module. The simulated usage profile, (i.e. users distribution among modules and number of events per hour per user), was based on a workload analysis of selected QC customers, representing three deployment sizes. The load profile used for this benchmark is based on the eight main QC modules, and the 39 most common and important business processes. For this benchmark, a total of 137 transactions had their response time measured. Benchmark tests included QC V10.0 new features and data, therefore all projects were VC (Version Control) enabled, and ERI activities were emulated. Some of the new features were not emulated as they run periodically and not on a constant level (e.g. Cross Project analysis, sharing and baseline). Performance may vary according to usage profile. The parameters which impact the load level are the number of concurrent users and the number of hits per second. These are reached by defining how many users will simulate each type of business process and in what frequency. The table below summarizes the activity level on each module. To view the load profile details, refer to the Tested Profile table in the Appendix section. Module Small Env. 500 Users Event s/h Users Medium Env Users Events/ H Users Large Env Users Events/H Users 500 Users Full Load For Recommended Env Events/H Users Defects Graphs Single Project Graphs Cross Project Requirements Test Lab Test Plan Storage ERI Global Total

10 3.4 Data Profile Data Profile Overview The benchmark Master DB consists of 10 large identical projects, against which all virtual users are running. The volume and complexity of the large projects was designed according to the DB analysis of selected QC customers Data Profile Structure To validate most accurate response time from benchmark testing, it is important to build QC projects which contain real-world data and structure. The project structure is based on selected QC customers. To ensure real-world conditions, each QC entity was tested using a specific profile. To view the load profile details, refer to the Data Profile by Entities table in the Appendix section Data Profile Sizing Regardless of the database QC projects structure, the transaction response time is also impacted by the database volume. Master DB size is approximately 11GB s per QC project (not including the file repository). To view the DB details, refer to the Data Profile by Table Size table in the Appendix section.

11 4 Recommended Environment Results 4.1 Recommended Environment Configuration QC V10.0 deployment has several recommended environments as defined by the QC product management Four recommended environments have passed full performance benchmark. Recommended environments were tested with full load of 500 concurrent users. Full load describes a scenario emulating more than 500% load of real-life environment at peak time (based on a workload analysis of selected QC customers), in terms of Events per hour, throughput and Hits per seconds. The table below summarizes the configuration used for the recommended environment check. Env Environment Configuration 500 Users Full Load For Recommended Env App: JBOSS on Win2003 DB: MSSQL 2005 App: WebLogic on Red Hat Linux DB: Oracle 10 App: JBOSS on HP-UX DB: Oracle 10 App: WebLogic on Sun Solaris DB: Oracle Transactions response time distribution Four of the recommended environments that were tested, achieved a satisfying average transaction response time and good resources management over executions and load profile changes (CPU, memory and I/O resources). The benchmark ran on multiple environment configurations chosen for their prevalence in today s QC customer install base. The scenario results are not meant to serve as basis of extrapolating suitability of one configuration over another because of various variables (for example, hardware, operating system, Application Server, DB type and DB system parameters).

12 Total of transaction types for which their response time was measured: 137 The average transaction response time is as follows: Configuration Transactions with response time < 1 sec Transactions with response time 1 3 sec Transactions with response time 3 10 sec Transactions with response time >10 sec JBOSS on Win % 27.7% 9.5% 3.6% WebLogic on Red Hat Linux 59.1% 21.2% 18.2% 1.5% JBOSS on HP UX 57.7% 21.9% 19.5% 1.5% WebLogic on Sun Solaris 47.4% 21.2% 24.8% 6.6% 4.3 System Resources Summary Table Configuration Env Size Users Hits / Sec Machine CPU % DB Disk Time JVM Used Memory HP UX Small Linux Small Win 2003 Small Solaris Small App Node DB App Node DB App Node DB App Node DB

13 5 Sizing / Scalability 5.1 Sizing/Scalability Configuration The environments described in the table bellow, took part of the QC scalability and sizing benchmark. Three QC deployment environments were tested, including small, medium and large. The DB size and DB server strength were constant throughout the process. The changed parameters were the amount of users, hits per second, and the amount of QC nodes. Results of this test provide customers with an option to map their deployment setup to one of the setups below. They can also get an idea as to the number of QC servers needed for deployment. QC nodes in each sizing category were identical. There was no comparison between the CPU strength and the amount of RAM. While testing all scenarios, a low level of CPU and memory utilization was maintained in order to take into consideration the fact that the benchmark was performed on a lab environment and not a production environment. The load simulated for sizing and scalability was of a real-life environment at peak time (based on a workload analysis of selected QC customers), in terms of events per hour, throughput and hits per seconds. Scalability and sizing scenarios started out with 500 users. It showed a good performance. Scenarios were then scaled up to 1000 users, using the same QC node (without adding additional CPU or memory). Third stage was scaled out by adding an identical QC node, and by increasing the load to 2500 users. Env Environment Configuration App: JBOSS on Win2003 DB: MSSQL 2005 App: WebLogic on Red Hat Linux DB: Oracle 10 App: JBOSS on HP-UX DB: Oracle 10 Small Env. Nodes:1 Users:500 Medium Env. Nodes:1 Users:1000 Large Env. Nodes:2 Users:2500

14 5.2 Sizing/Scalability Results All three environments including the three sizing scenarios, achieved a satisfying average transaction response time, and good resources management over executions and load profile changes (CPU, memory and I/O resources). The following graphs present the average overall response time (Y-Axis). The Application Server CPU and DB CPU utilization (Y-Axis) are shown for each environment and sizing configuration (amount of users on X-Axis). Analyzing the execution reports below, it can be affirmed that there are no significant degradations in response time and resources between the different types of environments. QC can be both scaled up to maximum amount of ~1000 users, and scaled out to run on multiple QC servers with minimum impact to performance. The results are accurate for the hardware used in the benchmark scenarios, as described in section 3. Graph Sizing WebLogic on Linux env CPU Utilization (%) Amount of Users DB CPU App Server CPU Overall TRT

15 Graph Sizing JBOSS on HP-UX env CPU Utilization (%) Amount of Users DB CPU App Server CPU Overall TRT Graph Sizing JBOSS on Win CPU Utilization (%) Amount of Users DB CPU App Server CPU Overall TRT

16 5.3 System Resources Summary Table Configuration Env Size Users Hits / Sec Machine CPU Small App Node 1 DB % DB Disk Time JVM Used Memory 221 HP UX Linux Windows Medium Large Small Medium Large Small Medium App Node 1 DB App Node 1 App Node 2 DB App Node DB App Node DB App Node App Node DB App Node DB App Node DB

17 6 JVM Tuning 6.1 General As part of the benchmark, and previous to all load executions, JVMs were tuned for optimal performance, ensuring the most accurate memory management on the QC server side. JVM configuration was tuned to improve performance in the benchmark environment. Customers can use the JVM configuration with the initial QC setup. However it should be finetuned during the initial stages of deployment as JVM activity may vary depending on the environment configuration (operating system, topology, and hardware) and QC usage. In addition, several JVM vendors were benchmarked for the different environments, in order to execute the tests with suitable JVMs (according to operating system, and the application server). All benchmarked environments were tested with a 64-bit operating system and a 32-bit JVM. 6.2 Tuning Sun JVM on JBOSS The following are the recommendations for JVM tuning. The JVM tuning was executed on Win Server 2003 Enterprise edition 64-bit, with JBOSS as the application server. All tests have been done with our full load in the background. 1. JVM parameters checked: Heap size Young generation sizing GC algorithms GC paralleling Permanent size 2. Parameters defining test behavior: Application server CPU Used heap Average freed memory by GC Accumulated GC pauses Throughput (Time the application is not in GC) Number of GC events 3. JVM optimal options for java command line set JAVA_OPTS=-Xms1400m -Xmx1400m -XX:+UseParNewGC - XX:PermSize=64m -XX:MaxPermSize=64m -XX:NewSize=412m - XX:MaxNewSize=412m

18 6.3 Additional JVMs JVMs were chosen for different benchmark executions based on vendor support, operating system, and application servers Weblogic on Linux Red Hat The JVM used for Linux environment with Weblogic 9.2 application server, was Oracle (BEA) vendor: BEA JRockit JVM (JRR) MEM_ARGS="-Xms1400m -Xmx1400m -Xns400m -Xgc:gencon - XXcompactRatio:5" WebLogic on Sun Solaris According to vendor (BEA), the only 32-bit JVM supported on Solaris 10 is from Sun. s92/92_over/overview.html Therefore, the JVM used for benchmarking was the 32-bit Sun JVM with the following options: MEM_ARGS="-Xms1400m -Xmx1400m -XX:+UseParNewGC JBOSS on HP-UX HP has released a specific JVM which is more suitable. It provides better performance with the HP-UX architecture. umber=hpuxjdkjre50 We got better response time with the HP-UX Sun JVM than with the standard Sun JVM. set JAVA_OPTS=-Xms1400m -Xmx1400m -XX:+UseParNewGC - XX:PermSize=64m -XX:MaxPermSize=64m -XX:NewSize=412m - XX:MaxNewSize=412m

19 Appendix

20 I. Tested Profile The following table details the load profile used per QC module. Module Defect Business Process Small Env 500 Users Events/H Users Pacing (sec) Medium Env 1000 Users Events/H Users Pacing (sec) Large Env 2500 Users Events/H Users Pacing (sec) Advanced filter Create/Delete Create Link from defect Filters + Audit Filter Multi-Value Group Grid Text Search Modify Update Multi-Value Global Login/Logout Defects Summary Graphs Req. Test Lab Test Plan Up / Down Defects Age Defects Trend TestSet Summary TestSet Progress Alert On Delete Coverage Analysis Create/Delete Expand Tree Filter hierarchy Group By Filter Traceability Update Requirement Add Folder and TestSet with Tests Instance Run Manual Test Show linked defects Create/Delete Subject Create/Delete Test+ Design Steps Expand Subject Filter Tests Update Test TEP_SmallCompare Upload Multiple File Upload Single Files Download Multi File Download 1 File Attachments ERI_Dependencies ERI ERI_Download Total

21 II. Data Profile by Entities The following table details the profile used for each QC entity. Module Entity Amount Structure Releases Requirements Business Components Test Plan Test Lab Release Folders 36 Releases 200 Release Cycles 2000 Requirements Component Folder 107 Component 5210 Component Step All Lists Tests BP Test to Component BP Parameter Cycle Folder 2802 Cycle Root Folder -> 5 F -> 2 F -> 10 Rel - > 10 Release Cycle 1 Root Req -> 30 Req -> [2 Req (9-10 Levels)] 1 Root Folder -> 100 F -> 50 Components -> 100 Steps 1 Root Folder -> 70 F 1 Root Folder -> 19 F -> [2 F, 4 Tests (9 Levels)] -> 5 Tests 1 Root Folder -> 50 F -> 20 Tests 1 Root Folder -> 50 F -> 20 Tests -> 20 Components 1 Root Folder -> 50 F -> 20 Tests -> 20 Components -> 10 Params 1 Root Folder -> 50 F -> 5 F -> 2 F -> 4 F -> 10 Tests Sets -> 20 Tests Run Defects Defects ERI Assets Relation Assets Repository items Defects 1 Root Folder -> 10 Res Folder -> 2 Res Folder -> 5RF -> 5 RF -> 2 RF -> 9 Files (5 Lib,3 Object Rep, 1 Recovery scenario)

22 III. Data Profile by Table Size The following table details the profile used by DB table size. Table Name Records count Table Name Records count ACTIONS 176 LINK ALERT MAILCOND 2 ALL_LISTS MODULES 7 ANALYSIS_ITEMS 5 POLICY_ENFORCEMENT_STATUS 1 ANALYSIS_ITEM_FOLDERS 1 RBT_CUSTOMIZATION 4 ASSET_RELATIONS RBT_CUSTOMIZATION_ANSWERS 36 ASSET_REPOSITORY_ITEMS RBT_CUSTOMIZATION_QUESTIONS 12 AUDIT_LOG RELEASES 200 AUDIT_PROPERTIES RELEASE_CYCLES 2000 BPTA_CHANGE_STATUS 4 RELEASE_FOLDERS 36 BPTEST_TO_COMPONENTS REQ BP_ITERATION REQ_COVER BP_ITER_PARAM REQ_CYCLES BP_PARAM REQ_RELEASES BP_STEP_PARAM REQ_TRACE 6000 BUG REQ_TYPE 6 BUG_MULTIVALUE REQ_TYPE_FIELD 292 BUG_TOKENS REQ_TYPE_HIER_RULES 6 COMMON_SETTINGS RESOURCES 9000 COMPONENT 5210 RESOURCE_FOLDERS 1631 COMPONENT_FOLDER 107 RULES 6 COMPONENT_STEP RUN CROS_REF RUNTIME_PARAM 42 CYCLE SEQUENCES 85 CYCL_FOLD 2802 STEP DATACONST 25 STEP_PARAMS DESSTEPS SYSTEM_FIELD 1082 FRAMEWORK_PARAM TABLES 76 GROUPS 5 TASK_LOG_TYPE 3 HIST_ASSET_RELATIONS TASK_STATE 6 HIST_ASSET_REPOSITORY_ITEMS TEST HIST_CROS_REF TESTCYCL HIST_DESSTEPS TEST_PARAMS HIST_STEP_PARAMS TOKENS 1156 HIST_TEST TRAN_RULES 9 HIST_TEST_PARAMS USERS 3001 HIST_USER_ASSETS USER_ASSETS LIBRARY_FOLDERS 1