Analysing Performance Testing Results Release 3.1

Transcription

1 Analysing Performance Testing Results Release StressTester Analysing Performance Testing Results 3.1

2 Copyright Copyright 2004, 2005, 2006 Reflective Solutions Ltd. All rights Reserved Restricted Rights Legend This software and documentation is subject to and made available only pursuant to the terms and conditions of the Reflective Solutions Ltd. license agreement and may be used or copied only in accordance with the terms of that agreement. It is against the law to copy the software except as specifically allowed in the agreement. This document may not, in whole or in part, be copied, photocopied, reproduced, translated or reduced to any electronic medium or machine-readable form without prior consent, in writing, from Reflective Solutions Ltd. Information in this document is subject to change without notice and does not represent a commitment on the part of Reflective Solutions Ltd. THE SOFTWARE AND DOCUMENTATION ARE PROVIDED AS IS WITHOUT WARRANTY OF ANY KIND INCLUDING WITHOUT LIMITIATION, ANY WARRANTY OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. FURTHER REFLECTIVE SOLUTIONS LTD. DOES NOT WARRANT, GUARANTEE OR MAKE ANY REPRESENTATIONS REGARDING THE USE OR THE RESULTS OF THE USE, OF THE SOFTWARE OR WRITTEN MATERIAL IN TERMS OF CORRECTNESS, ACCURACY, RELIABILITY OR OTHERWISE. Trademarks or Service Marks jtune and StressTester are registered trademarks of Reflective Solutions Ltd. All other trademarks are the property of their respective companies. 2 StressTester Analysing Performance Testing Results 3.1

3 Contents Copyright...1 Copyright...2 Restricted Rights Legend...2 Trademarks or Service Marks...2 Introduction...4 Overview of Architecture...4 Injectors...4 StressTester Database...5 StressTester GUI...5 StressTester GUI Conventions...6 Mandatory Fields...6 Read-Only Fields...6 Access to Buttons...6 Time Fields...6 Field Grouping...7 Tables...7 User Journey Level Analysis...8 The User Journey Results Screen...8 Control Area...9 Graph Area...10 Results Summary...11 Step level Analysis...12 Viewing Step Level Results...12 Graph Area...13 Results Summary...13 Customising Graphs...15 Graph Definition Basics...16 Defining a Single Line Graph...17 Defining a Single Line Timeline Graph...17 Defining a Multi Line Timeline Graph...18 Defining a Histogram Graph...19 Defining a Pie Chart Graph...19 Graph SQL...20 Examples...21 Customising Data Tables...22 Data Table Definition...23 Data Table SQL...24 Examples...24 Troubleshooting Problems...25 Error Messages...25 Java Consoles...25 Injector Log and Error File...25 Reflective Solutions Support...26 Appendix A...27 Java Interfaces...27 The ResultProcessor Interface...27 The DataConverter Interface StressTester Analysing Performance Testing Results 3.1

4 Introduction StressTester is an enterprise level performance testing tool that allows users to place an application under correct, realistic and repeatable load and to measure how the application performs under that load. The load can be generated from multiple locations both within the organisation s network and from anywhere on the Internet. By using StressTester, organisations can easily understand how their applications will perform under realistic and correct load, identify possible performance and scalability problems and, using StressTester s Resource Monitor feature, identify the underlying resources that are causing the performance problems. As StressTester allows the data supplied to an application to be varied and the response from the application to be checked to ensure the application is functioning correctly, StressTester can also be used to functionally test applications. Adoption for this reason is shown to reduce project timescales, effort required and hence overall development costs. Overview of Architecture StressTester consists of three components: StressTester Injectors The StressTester Database The StressTester graphical user interface ( StressTester GUI ). Injectors StressTester Injectors are the processes that simulate user load against an application. Each Injector is a long-running process; once it has been started it can either be idle or participating in a performance test. The process only dies when it is killed or the machine that it runs upon is stopped this means that Injectors can be started once and thereafter require no management. In fact, it is common for Injectors to be started and managed by the operating system on Windows this is known as a service, on UNIX/Linux a start-up process. Injectors can actually spawn sub-processes to assist them in creating large loads. These sub-processes are automatically managed by the Injector and 4 StressTester Analysing Performance Testing Results 3.1

5 when they are no longer required (i.e. the performance test finishes) the Injector will automatically stop the process. StressTester Database The StressTester database contains details of all the application transactions that are to be simulated, the underlying resources to be measured and the results obtained by the Injectors. This database is documented allowing users to analyse the data using any tool that is SQL compliant StressTester GUI The StressTester GUI provides users with a single interface for the creation and editing of application user journeys, and the creation and execution of performance test runs. In addition the GUI provides a comprehensive and fully customisable analysis engine allowing the user to easily understand the results of a performance test. A powerful feature of the GUI is the ability of users to add their own graphs or tables to the user interface any data that is held within the StressTester database can be displayed in a line, multi-line, timeline, histogram or pie chart graph or summarised within a data table. 5 StressTester Analysing Performance Testing Results 3.1

6 StressTester GUI Conventions The StressTester GUI has been designed to have a standard look and feel and common functionally. The intention is that once the workings of the GUI are understood on one screen, all other screens are familiar and easy to use from the user s perspective. The sections below describe the standard look and feel and functionality of the StressTester GUI. Mandatory Fields All mandatory fields have a yellow background. Read-Only Fields All read-only fields are greyed-out and the cursor cannot be placed within the field. Access to Buttons Screen buttons are normally greyed out until all required mandatory fields have been completed. Hence if you cannot gain access to a button, the first course of action should be to confirm that all mandatory fields are completed correctly. Time Fields All fields that accept time values indicate in the field name the units that should be used. (sec) indicates times should be specified in seconds and (ms) that they should be specified in milliseconds. 6 StressTester Analysing Performance Testing Results 3.1

7 Field Grouping Related Screen Fields are grouped within boxes with a title at the top of the box indicating the purpose of the fields. Tables Nearly all table columns are sortable; clicking the column heading sorts in ascending order and clicking again reverses the sort. Table columns can be repositioned by dragging the column heading to the required position. Column widths can be resized by dragging the edge of the column heading as required. 7 StressTester Analysing Performance Testing Results 3.1

8 User Journey Level Analysis The results of a performance test are analysed using the StressTester GUI. It is possible to access the StressTester database directly using an appropriate SQL tool but the facilities and ease-of-configuration of the StressTester GUI mean you should never have to do so. The StressTester GUI allows performance test results to be analysed at two levels: User Journey and Step. A User Journey represents a simulated usage of the application or a Resource Monitor that has measured an underlying system under test resource. A User Journey simulating an application usage may consist of many Steps whereas a Resource Monitor only ever has one Step. Each User Journey represents one interaction between the application front end and the application server. For example, a web page may consist of HTML, cascading style sheets, JavaScript and images. Each of these is an individual request to the application and hence an individual Step. The User Journey Results Screen User Journey results are viewed by selecting the View Test Results button or by selecting the Performance Tests menu and then the View Test Results option. A screen similar to that below will be shown. 8 StressTester Analysing Performance Testing Results 3.1

9 The screen has three areas a control area and two results analysis areas: one in graphical format and the other in tabular data format. The analysis areas are configured using property files (see subsequent sections) and hence the user has full control over what information is displayed and in what format. Control Area The control area of the screen contains the following fields: Field Name Description Valid Values Performance Test Drop down list of all performance tests (past and currently executing). As selected Refresh Interval (sec) Whilst a test is running, the Injectors are continuously updating the StressTester database with details of the performance of the application under test. Numeric This interval defines the time 9 StressTester Analysing Performance Testing Results 3.1

10 Field Name Description Valid Values in seconds between successive refreshes of the analysis areas of the screen. If this value is set to 0, the refresh will not occur. Start Date End Date User Journey / Monitor Selection The information displayed in the analysis areas of the screen can be restricted to certain date and time ranges. The tokens NOW and TODAY may be used to represent current time or day (midnight). It is possible to use NOW and TODAY with simple arithmetic operators as follows: -xd (x days ago) -xh (x hours ago) -xm (x mins ago) Hence a start date of NOW- 2h means a start date of two hours ago. The tree navigation is used to select the User Journey and Monitors to display in the analysis areas. A legal date/time in the format of DD/MM/YYYY or DD/MM/YYYY HH:MM As selected The analysis areas may be refreshed at any time by clicking the Refresh Now button. Graph Area The graph area displays the data selected in the control area in a number of graphical formats. The available graphs are represented by tabs at the top of this area. When a tab is clicked, the graph it represents is displayed. 10 StressTester Analysing Performance Testing Results 3.1

11 The screen shot below shows an example when the second tab ( 1 Min Avg response ) has been selected. Graphs can be customised by right-clicking on the graph and then selecting the Properties option. Graphs can be saved to an external file by right-clicking on the graph and selecting the Save As option. Results Summary The results summary area displays the data selected in the control area in a number of data tables. The available tables are represented by tabs at the bottom of this area. When a tab is clicked, the table it represents is displayed. The table may be sorted by clicking the column headings the first click sorts the table in ascending order based on this column, a subsequent click sorts the table in descending order. 11 StressTester Analysing Performance Testing Results 3.1

12 Step level Analysis User Journey level analysis allows User Journeys to be compared alongside each other, and alongside the results of Resource Monitors. When a User Journey is seen to have performance problems, drilling down to the Step level makes it possible to identify the application requests that have long response times or errors. Hence Step level analysis allows fine grained performance and error investigation to be performed. Viewing Step Level Results To further investigate the response times of a User Journey, double click the User Journey name within the results summary area of the User Journey analysis screen. The Step level analysis screen will be displayed. This screen resembles the User Journey analysis screen and has the same three areas. However, the graphs and data displayed within this screen are relevant to just the selected User Journey and hence this screen allows detailed analysis of a User Journey s performance. The screen shot below shows an example of this screen with all the User Journey Steps selected. 12 StressTester Analysing Performance Testing Results 3.1

13 Graph Area The graph area displays the data selected in the control area in a number of graphical formats. The available graphs are represented by tabs at the top of this area. When a tab is clicked, the graph it represents is displayed. Graphs can be customised by right-clicking on the graph and then selecting the Properties option. Graphs can be saved by right-clicking on the graph and selecting the Save As option. Results Summary The results summary area displays the data selected in the control area in a number of data tables. The available tables are represented by tabs at the bottom of this area. When a tab is clicked, the table it represents is displayed. 13 StressTester Analysing Performance Testing Results 3.1

14 The Stop Test button can be clicked at any time to stop the performance test. 14 StressTester Analysing Performance Testing Results 3.1

15 Customising Graphs A major benefit of the StressTester GUI is that the user can customise the number and type of graphs displayed within the results analysis screens. This allows the user to represent the data they are interested in, in the format that best suits them, and means that they are in control of the analysis as opposed to the tool dictating the user s process. The graphs displayed within the StressTester GUI are configured using property files that are stored within the properties directory (and further subdirectories) within the StressTester GUI installation directory. The file graph.script.properties controls the graphs that will appear within the User Journey analysis screen and the graph.step.properties the graphs that will appear within the Step analysis screen. Each file consists of one or more name-value pairs separated by the equals sign. The name of each pair has the format: graph<numeric>.properties where <numeric> identifies the position the graph will appear in the graph tabs on the screen (the first graph has a position of 1). The value of each pair is the name of a properties file that exists within the properties/graph directory under the installation directory. The file specified defines the graph. An example entry is as follows: graph1.properties=avgresp_users_script.properties There are five types of graph that can be defined: single line, single timeline, multi timeline, histogram and pie chart. 15 StressTester Analysing Performance Testing Results 3.1

16 Graph Definition Basics All graph definition files, irrespective of the graph type, share some common features. The following lists the common properties that may be specified in any graph definition file. All properties are mandatory unless otherwise specified: Property Name graphtype Description Specifies the type of graph to be displayed. Valid values are numbersingleline, timesingleline, timemultiline, histogram and piechart tabtitle tabtooltip graphtitle converterclass [optional] The labelling that will be displayed on the graph s tab in the StressTester GUI. The tool tip that will be displayed if the mouse is placed over the graph s tab. The title that will be displayed on the graph. (This can be subsequently modified at run time by right-clicking within the graph and selecting the Properties option.) The key of a name-value pair within the property file: typestoclasses.properties. The value of the pair is a Java class that processes the results of the SQL statements (see below) prior to the graphs being drawn. Processing can be performed by any class that implements the following Java interface: com.reflective.stresstester.gui. graph.base.resultprocessor See Appendix A for a description of this Interface. 16 StressTester Analysing Performance Testing Results 3.1

17 Defining a Single Line Graph A Single Line graph will draw one line on the graph for every User Journey or Step selected within the control area. The following properties define the single line timeline graph. All properties are mandatory unless otherwise specified: Property Name axisxtitle axisy1title axisy1sql Description The title of the X axis. (This can be subsequently modified at run time by right-clicking within the graph and selecting the Properties option.) The title of the left-hand side Y axis. (This can be subsequently modified at run time by right-clicking within the graph and selecting the Properties option.) The SQL that will return the data that is to be graphed Defining a Single Line Timeline Graph A Single Line Timeline graph will draw one line on the graph for every User Journey or Step selected within the control area. The X axis always represents time. The following properties define the single line timeline graph. All properties are mandatory unless otherwise specified: Property Name axisxtitle axisy1title axisy1sql Description The title of the X axis. (This can be subsequently modified at run time by right-clicking within the graph and selecting the Properties option.) The title of the left-hand side Y axis. (This can be subsequently modified at run time by right-clicking within the graph and selecting the Properties option.) The SQL that will return the data that is to be graphed. 17 StressTester Analysing Performance Testing Results 3.1

18 Defining a Multi Line Timeline Graph A Multi Line Timeline graph will draw one line against the left Y axis for every User Journey or Step selected within the Control area, and a further line against the right Y axis the representation of which depends on the graph configuration. The X axis always represents time. The following properties define the multi line timeline graph. All properties are mandatory unless otherwise specified: Property Name axisxtitle axisy1title axisy1sql keylabely1 axisy2title axisy2sql axisy2type keylabely2 Description The title of the X axis. (This can be subsequently modified at run time by right-clicking within the graph and selecting the Properties option.) The title of the left-hand side Y axis. (This can be subsequently modified at run time by right-clicking within the graph and selecting the Properties option.) The SQL that will return the data that is to be graphed against the left Y axis. When a mouse pointer is hovered over a point on a graph, the legend presented comprises of the User Journey or Step name followed by this value in braces to identify the measurement type. The final segment of the legend is the time and date and the actual measured value. The title of the right Y axis. The SQL that will return the data that is to be graphed against the right Y axis. The type that should be applied to the Y2 axis labels. When a mouse pointer is hovered over a point on a graph, the legend presented comprises of the script or step name followed by this value in braces to identify the measurement type. The final segment of the legend is the time and date and the actual measured value. 18 StressTester Analysing Performance Testing Results 3.1

19 Defining a Histogram Graph A Histogram graph will draw one bar for each unique value to be plotted against the X axis. The following properties define the histogram graph. All properties are mandatory unless otherwise specified: Property Name axisxtitle axisy1title axisy1sql Description The title of the X axis. (This can be subsequently modified at run time by right-clicking within the graph and selecting the Properties option.) The title of the left-hand side Y axis. (This can be subsequently modified at run time by right-clicking within the graph and selecting the Properties option.) The SQL that will return the data that is to be graphed. Defining a Pie Chart Graph A Pie Chart will draw one segment for each User Journey or Step selected in the control area. The following properties define the pie chart graph. All properties are mandatory unless otherwise specified: Property Name axisy1sql Description The SQL that will return the data that is to be graphed. The SQL to return the data for a pie chart should select two columns the first always has a numeric value of 1, the second column contains the data to be plotted. 19 StressTester Analysing Performance Testing Results 3.1

20 Graph SQL The previous sections highlight that the main configuration property when creating a new graph is the SQL statement to select the data to be plotted. All SQL property names may be extended by the addition of one of the following suffixes:.oracle.sqlserver.mysql. These suffixes allow one graph definition file to be used against more than one database type and further, to be able to use non-standard SQL functions supported by a particular DBMS. The StressTester GUI first searches for the required property with the appropriate database extension, if this property is not found it will use the property without any database extension. Graph SQL can include placeholders that the StressTester GUI will replace at runtime to ensure the correct data is selected from the database. The following placeholders are available: Placeholder {RUN_ID} {SCRIPT_IDS} {STEP_IDS} Description This will be replaced by the selected performance test identifier. It should be specified in all SQL statements. This will be replaced by all selected User Journey identifiers. StressTester replaces this token with a list in the form of (id1, id2,..) so it is ideal to use this token with the SQL IN operator. This will be replaced by all selected Step identifiers. StressTester replaces this token with a list in the form of (id1, id2,..) so it is ideal to use this token with the SQL IN descriptor. This token is usually specified in SQL statements that wish to display data at Step level. {START_DATE} This will be replaced by the entered or calculated start date to restrict the data that is returned. {END_DATE} This will be replaced by the entered or calculated end date to restrict the data that is returned. 20 StressTester Analysing Performance Testing Results 3.1

21 Examples The StressTester GUI is supplied configured with a standard graph set that provides examples of valid graph configurations. By analysing the graph property files and referring to the above, the user should gain a rapid understanding of how the custom graphing facility works and be able to quickly extend and customise the supplied graphs. 21 StressTester Analysing Performance Testing Results 3.1

22 Customising Data Tables As well as customisable graphs, the user has full control over the number, type and content of the data tables displayed within the two analysis screens. This allows the user to represent the data they are interested in, in the format that best suites them and means that they are in control of the analysis as opposed to the tool dictating the user s process. The data tables displayed within the StressTester GUI are configured using property files that are stored within the properties directory (and further subdirectories) within the StressTester GUI installation directory. The file table.script.properties controls the data tables that will appear within the User Journey analysis screen, and the file table.step.properties controls the data tables that will appear within the Step analysis screen. Each file consists of one or more name-value pairs separated by the equals sign. The name of each pair has the format table<numeric>.properties where <numeric> identifies the position the data tables will appear in the data tables tabs on the screen. The value of each pair is the name of a file that exists within the properties/table directory under the installation directory. The file specified defines the data table rows, columns, column names and values. An example entry is as follows: table1.properties= scriptsummary.properties 22 StressTester Analysing Performance Testing Results 3.1

23 Data Table Definition The following properties can be specified within a data table property file. All properties are mandatory unless otherwise specified: Property Name tabtitle tabtooltip converterclass Description The text presented on the data table tab. The text presented when the user hovers their mouse over the tab. The full name of a Java class executed immediately after the SQL to populate the data table is executed to pre-process the result set prior to presentation by the GUI. The objective of this mechanism is to be able to handle custom calculations (e.g. Standard Deviation, PCurve etc) that are not possible using SQL. Processing can be performed by any class that implements the following Java interface: com.reflective.stresstester.gui. result.converter.dataconverter See Appendix A for a description of this Interface column<n>.name column<n>.sql SummaryReq Summary.column<n> The title to be shown at the head of the identified column. Column numbering starts from 1. The SQL to return the initial values (prior to any converterclass processing) for each column of the table. Y or N indicating whether a data table summary row is required. Describes how each column summary value is to be computed. Valid values are: Total sum the column Max the highest value Min the lowest value Avg average the column values None leave the cell blank 23 StressTester Analysing Performance Testing Results 3.1

24 Data Table SQL The same rules and guidelines refer to creating data table SQL as for creating graph SQL. Please refer to the Graph SQL section above for details of how to create the data table SQL. Examples The StressTester GUI is supplied configured with a standard data table set that provides examples of valid data table configurations. By analysing the data table property files and referring to the above, the user should gain a rapid understanding of how the custom facility works and be able to quickly extend and customise the supplied data tables. 24 StressTester Analysing Performance Testing Results 3.1

25 Troubleshooting Problems When a problem occurs, or when StressTester reports an unexpected error, the following resources should be used to understand and rectify the problem. Error Messages Reflective Solutions has attempted to make the product error messages as understandable and clear as possible. Therefore, any error messages reported by the product should be investigated to see whether they provide information as to why a particular problem may be occurring. Java Consoles The StressTester GUI and Injector are written in Java and hence run within a JVM. The JVM has an associated operating system process and by default writes information and error messages to the standard output and standard error destinations. Both of these should be investigated to see if there are any underlying JVM problems that are causing the problem to occur. Injector Log and Error File Each Injector creates at least a log file and, when errors occur, an error file. These are located in the directory specified in the Injector configuration and should be checked for errors if unexpected problems occur. 25 StressTester Analysing Performance Testing Results 3.1

26 Reflective Solutions Support Having followed the instructions above and having gathered the information detailed in each section, you are in a position to call Reflective Solutions Support if you still cannot rectify the problem yourself. Reflective Solutions Support can be contacted via the following methods: By Phone: By support@reflective.com 26 StressTester Analysing Performance Testing Results 3.1

27 Appendix A Java Interfaces The ResultProcessor Interface package com.reflective.stresstester.gui.graph.base; /** * Once a set of results have been obtained from the * database, it is possible to process them before * they are displayed in the graph on the analysis screens. * * Classes that manipulate the data in such a manner * must implement this interface. * * The implementing class needs to be added to the graph * properties file with the key 'converterclass'. */ public interface ResultProcessor { } /** * This method takes in a Collection of Object arrays. * Each array is the equivalent of a point on the graph. * The Object[] is the results of the query, storing the * values in the order queried. * datain Collection of Object[]. Each Object[] * represents a point on the graph Collection of Object[] with data to go to the * graph */ public Collection<Object[]> processresults(collection<object[]> datain); 27 StressTester Analysing Performance Testing Results 3.1

28 The DataConverter Interface package com.reflective.stresstester.gui.result.converter; /** * Once a set of results have been obtained from the * database, it is possible to do processing on them before * they are displayed in the table on the results screen. * * Classes that manipulate the data in such a manner * must implement this interface. * * The implementing class needs to be added to the table * properties file with the key 'converterclass'. */ public interface DataConverter { /** * This method takes in a Collection of Object arrays. * Each array is the equivalent of a row in the table, * representing either a monitor, a user journey or a * step depending on where this class is being used. The * Object[] is the results of the query, storing the * values in the order queried. * input Collection of Object[]. Each Table row is * Object[] object in collection. Collection of Object[] with data to go to * Table */ public Collection<Object[]> convertdata(final Collection<Object[]> input); } 28 StressTester Analysing Performance Testing Results 3.1