SOFTWARE PERFORMANCE TESTING TIPS WITH JMETER

Transcription

1 SOFTWARE PERFORMANCE TESTING TIPS WITH JMETER

2 When you hear the term software performance testing, what comes to mind? How many users can my system handle? How many users can it handle if I want to maintain better than a two-second response time? What will be the response time if 000 users try to print a report at the same time? Does the performance of my software degrade over time? Whether you ve got a cloud-based SaaS (software-as-a-service) solution or a premises-based client server system, you probably want the answer before you release the software to your users. SETTING YOUR PERFORMANCE TESTING OBJECTIVES These types of questions are the root of software performance testing and help you to determine what your goals are. Based on our experience in performance testing over the last 0 years, we ve assembled a few common objectives:» Compare two application versions to ascertain improvement» Measure software performance with conditions replicating actual use» Identify performance issues to optimize user experience» Determine application performance and scalability as a function of normal and peak load conditions» Benchmark third-party products to determine which integrate best with your software WHO S ON FIRST? But as the basis for your objectives, you ll want to figure out what your users are really doing. Maybe 000 users printing a report at the same time is not a realistic scenario. Perhaps, you ll have some users printing a report while other users are logging in and some are doing a frequent search. Figuring out your user profiles is one of the key foundations in doing performance testing. What you ll find is that your users are doing so many different workflows and scenarios, some of which you expect, like accessing reports or logging in. However, other scenarios may be unexpected such as executing a billing reconciliation or entering nurse notes that are then checked by a doctor or physician assistant. Regardless, you ll have to prioritize your performance testing. There are numerous ways to do this. You can do a detailed analysis of your users paths through your software with Google Analytics or other equivalent techniques, or you can use a heuristic prioritization method where you use information that you know and prioritize based on the most likely user scenarios while keeping frequency and importance in mind.

3 FINDING THE RIGHT TOOL After you ve gotten all that figured out, you ll need to think about what tool to use. These days, there are many lightweight and open source tools that can do the job if you know what job you want them to do. Picking the right tool is important because not every tool works well with all software. These days, while most applications are web applications, not all web applications are the same and there are still many client-server applications with thick clients through software installed on the local machine other than a browser. Therefore, picking the right tool not only depends on your objectives and the types of performance information you want, but also the skills you have in house to implement the tool. One of the popular open source tools that we ve been using a lot lately is JMeter, which many say has a high labor quotient. We ve used dozens of software performance testing tools, including open source, commercial, light-weight and heavy-burdensome. While we wouldn t disagree that JMeter requires some effort to get meaningful and actionable results, performance testing with any tool is not easy. The remainder of this paper discusses some of the most common problems we run into when using JMeter. COMMON PROBLEMS WITH JMETER HANDLING SESSION IDS FOR A COMPLEX WEBSITE Some websites automatically generate a session ID for each user which can be used as a temporary credential to allow you to execute operations within a website. However, the session ID will expire as soon as you close your browser or after a certain amount of time. In other words, if you want to reuse the script (because different users will use the same script and you want to use the script when executing different test runs), you should use dynamic values instead of fixed values for the session ID. In that case we need to use a regular expression to handle the session ID. To do this, we usually add an HTTP Cookie Manager, when the session ID is included in the cookie. The HTTP Cookie Manger will automatically save the session ID value and use it for the following request. Figure shows example code to make this happen, while Figure 2 shows the solution. Figure. Example code for managing the session ID 2

4 Figure 2. Resolving the session ID Some websites automatically generate a session ID for each user which can be used as a temporary credential to allow you to execute operations within a website. However, the session ID will expire as soon as you close your browser or after a certain amount of time. In other words, if you want to reuse the script (because different users will use the same script and you want to use the script when executing different test runs), you should use dynamic values instead of fixed values for the session ID. In that case we need to use a regular expression to handle the session ID. To do this, we usually add an HTTP Cookie Manager, when the session ID is included in the cookie. The HTTP Cookie Manger will automatically save the session ID value and use it for the following request. Figure shows example code to make this happen, while Figure 2 shows the solution. Figure 3. Response data 3

5 In general, the way that we solve this problem is to use a JMeter internal function called Regular Expression Extractor to obtain the value of the session ID from Response data. Figure 4 shows an example of this. Figure 4. Using Regular Expression Extractor to obtain session ID The problem is that each situation is different because the returned data structure in Response data is different. We need to determine the proper regular expression to get the corresponding value based on each actual situation. 2 CORRELATION Correlation is one of several critical aspects in performance scripting. It generally includes fetching dynamic data from preceding requests/calls and posting it to the subsequent requests. To accomplish this, we use a regular expression to fetch the dynamic value and then correlate the value with the parameter to make sure it can be used in the following script calls. Figure 5 shows an example using the color class. ) For the color class, our goal is to extract both ID and name for the following requests. Figure 5. Extracting the ID and name for the color class 4

6 2) We use regular expressions and get the following result: Figure 6. Using regular expressions JMeter provides a function plugin called Regular Expression Extractor, which automatically pulls the value from previous response data as long as a correct expression has been input in the Regular Expression Extractor. As shown in Figure 6, for instance, after inputting expression class="colors" id="(.+?)" name ="(.+?)" in the Regular Expression Extractor, you will get above values after running the scripts. These values are automatically returned by JMeter, but not all of them are useful. We only care about whether the ID and name, as shown in Figure 5, have been returned. For example, in the first row in Figure 5, we check the returned result in Figure 6 and check whether the value ID blu returned or not. What we see is that the value blu has returned in parameter colors g. 5

7 3) In our case, JMeter creates a variable called colors_matchnr during runtime, which basically returns the total number of matches in the response or page. The BeanShell correlates the value of the parameters. The following script showing in Figure 7 prints out the value that returned in Figure 6. For instance, as shown in step 4 in Figure 7, we are trying to print the value of colors g, which happens to be blu. Figure 7. BeanShell example 3 INCREASING LOAD AND ANALYZING PERFORMANCE METRICS This is the most important and often most difficult part when doing performance testing. Application issues can sometimes manifest themselves when you introduce load. There are numerous factors that influence the performance results you obtain with JMeter. For instance, if the application is designed by Java, it consumes more memory than some other development platforms (like C++, C#, etc.) because Java uses a garbage collection mechanism. Garbage collection, whose purpose is to identify and remove objects that are no longer needed by a program, is also known as automatic memory management since the Java virtual machine automatically removes the unused variables and objects from memory. A Java object is subject to garbage collection when it becomes unreachable to the program in which it is used. However, problems with the application will arise if the objects have not been removed correctly because these objects consume memory and the CPU. Effort should be made to remove objects that are not useful; otherwise, they will consume important resources. 6

8 Another factor that will influence the performance data result is that JMeter execution is highly influenced by the memory of the machine in which JMeter is installed. For example, a machine with a GB RAM will severely limit your thread execution. If we identify that the performance results are being influenced by insufficient resources like memory, then we need to verify that this is the specific reason, eliminate that influence, and run the tests again. Most times, it s not so easy. This is just one example illustrating that performance testing issues that need to be considered and troubleshot sometimes fall outside the application under test. When you are certain that your environment is not influencing the operation and results of JMeter, then you re ready to add load. Adding load is difficult because we need to adjust the load for different scenarios and in different combinations of scenarios to identify the performance issues of an application. It s like a patient that goes to see a doctor. The patient may complain to the doctor about their aches and pains. However, the doctor needs to ask the right questions to arrive at an accurate diagnosis for the patient. As the performance testing doctor, we need to do multiple performance tests with many scenarios multiple times in order to identify the application s issues. Regarding performance metrics, there are various kinds of metrics, such as response time, error (failed user requests) and throughput. Figure 8. Performance testing results example As shown in Figure 8, as the number of users continues to increase, you can see the degradation in performance. 7

9 SUMMARY In the end, you ll want to analyze performance results to determine where to put your effort in speeding up your software Just make sure that you treat your performance testing and subsequent improvements like a controlled experiment. If after analyzing your results you determine that a certain user scenario requires attention, make the necessary changes and copiously document those changes before going back and running your performance tests in exactly the same way in the same environment to determine if you ve improved. Changing more than one thing at a time won t give you the control to determine what worked and what didn t. found that even some of the simplest functions can be slower than complex algorithms due to many reasons, whether it s the software s architecture or its use of third-party plugins. When fine-tuning your application, remember that performance is what wins a race. When users are comparing your application against others, keep in mind that if the race were just about functionality and a pretty interface, numerous contenders would vie for the win. But only when the rubber hits the road and users ramp up, will you be the winner? You might also think that performance will most likely suffer where functions are complex, but you d be surprised. We ve 8

10 ABOUT XBOSOFT XBOSoft is a software testing and quality assurance company that offers services in web, mobile and desktop applications, as well as broad domain experience with extended expertise in healthcare, finance, and software security. XBOSoft s quality process assessments and test methodologies speed products to market and improve clients software quality and performance throughout the software s life cycle. With a proven track record working for Fortune 00 companies to small independent developers, XBOSoft today has over 00 employees in San Francisco and Beijing offices serving clients in the US and Europe. As the firm surpasses its tenth year of continuous growth, XBOSoft proudly offers its full range of testing and quality assurance as both onand off-shore services. CONTACT XBOSOFT PHONE: services@xbosoft.com

11