Intelligent Performance Software Testing

Size: px

Start display at page:

Download "Intelligent Performance Software Testing"

Baldwin Augustine Peters
5 years ago
Views:

1 White Paper Intelligent Performance Software Testing The field of software functional testing is undergoing a major transformation. What used to be an onerous manual process took a big step forward with the advent of Selenium and other testing tools. But those tools remain heavily developer-centric and are primarily tethered to rigid script-based approaches that do not scale. And now, with the rise of cloud, big data and machine learning, companies need to re-think their testing strategies and technologies yet again. The core concept behind this new approach is that, in this era of agile development and rapid customer response, testing can no longer be an adjunct to the development process: it needs to be an integral and strategic component of the workflow. Functionize is doing just that with the introduction of our ground-breaking scalable autonomous test platform. The driving force behind this platform is Adaptive Event Analysis, a patented and proprietary hybrid of algorithms which span supervised, unsupervised, reinforcement learning and computer vision, enabling lightning-fast test creation and execution, selfhealing maintenance, and actionable analytics. Following the execution of any functional test, Functionize provides extensive performance insights without any additional steps. Key features include benchmark rendering, latency measurements on web-tier services and back-end processes, and throughput analysis on payment and other external APIs. Since Functionize runs everything in a browser session, the performance data corresponds very closely with what customers actually experience.

2 Table of Contents What is Performance Testing? 3 Next Generation Performance Testing 3 An Example of Performance Metrics 4 Technical Page Load Speed Details 5 Functionize Provides Ultimate Performance 6

3 What is Performance Testing? Performance testing is a set of procedures designed to evaluate the responsiveness of website applications by measuring capacities and collecting analytical data, including timing thresholds, load maximums/minimums, and numerous other data points. All of these performance metric data points must be engineered and evaluated to derive the optimal operating conditions for the application, to achieve the highest level of responsiveness within acceptable tolerances of stability and scalability. Performance testing is an integral component of performance engineering, and the two often provide stimulus for mutual improvement. Every aspect of a web application which can be measured, such as asynchronous loading of SPA page content, may be subject to evaluation through testing. Performance testing includes a subset of specific test procedures each of which focuses on a measurable aspect of the application s operation. The different types of performance testing procedures that can be applied to evaluate a web application include: Load testing - evaluates how the app will behave under specific conditions such as a high volume of requests from a number of simultaneous users Stress testing - provides metrics which enable system administrators to determine the upper limits of request volume capacity, and makes possible realistic design considerations Spike testing - suddenly injects a high volume of simultaneous requests to the application and measures the resultant performance Soak testing - is closely related to stress testing and seeks to determine how long the app can endure these high volume requests Performance testing is essentially machine measuring the temporal behavior of the web application, which is precisely why automation tools are mandated. Now that a truly intelligent automation tool avails itself in the form of Functionize, it naturally follows that performance testing procedures and resulting analytics and metrics data are seamlessly integrated into one panel along with other crucial testing workflows, including functional and load testing. Next Generation Performance Testing Any test suite created within Functionize will automatically include in-depth performance analytics. As tests are executed, Functionize collects precise performance metric data, including proprietary metrics, which are visualized through charts and graphs, and are available for comparing versions throughout the testing cycle. One of the proprietary metrics is called Visual Completion. This feature combines computer vision and machine learning to measure precisely the time required to render the page visually complete for the user experience. This is a useful metric

4 for modeling, as it allows for understanding the user experience during high loads. Functionize s Test Cloud simulates this by spinning up multiple instances of an app UI on virtual machines and using virtual users to open the app pages in real browsers to fully model the timing thresholds of every aspect of a real user experience with your app, including the instant at which the page is fully rendered. An Example of Performance Metrics The below case study illustrates how Functionize aggregates performance data on an executed test case. In this particular case, Functionize is measuring performance data on a Sam s Club UI with a project to examine the Add TV to Cart page (see Screenshot 1. below). Leading up to this point, Functionize s Test Cloud has executed all recorded assertions in tandem and collected timing and duration data for every property and attribute, every input and output in the test plan. Looking closer at the screenshot below, note the comprehensive list of analytics data acquired for every page loaded in each project run. Included in the performance metrics is the file name, the time it loaded, and time required to load. A comparison chart makes it possible to quickly scan through a list of loaded elements for the test session and load times to determine which images and files loaded slowly, likely negatively impacting the user experience. Every detail of the chart such as color coding is organized to visualize the temporal components of a page load to present the critical statistics in an easy to read form. Screenshot 1. Performance Metrics Conventional visual measurements emphasize dominteractive, but Functionize measures and reports the visual completeness of each page load, which is indicative of the end user s perspective. Action duration details show a chart of all 16 steps in our example case study and exactly how much time was required to complete each step by Functionize s Test Cloud. Initializations, verifications, and every step of every test are rendered in an intuitive and interactive graph. This facilitates efficient assessments of the user experience and pinpoints any potential bottlenecks that might result from user traffic to the web application.

5 Screenshot 2. Performance Metrics (continued) Functionize executes performance testing on incredibly complex web apps with multiple Iframe elements, heavy JavaScript coding, and asynchronous callbacks. Creating performance tests in Functionize is the natural byproduct of any functional test creation since every assertion / page interaction recorded in a test step results in performance metric data collection when the test case is run. No additional scripting or customization is required. Attribute panels give the tester full access to tag names, text, visibility features, input values, and class attributes. Every action possible on a web app can be modeled and evaluated for performance by non-technical testers right in the Functionize control panel. Operators for evaluating value comparison, such as greater than or equal to, are available in English words rather than coding language symbols to empower non-technical testers to modify attributes without scripting. Technical Page Load Speed Details A further examination of Functionize s Page Load Speed metric set reveals further performance analytics. Beginning with Domain Lookup for the page, Functionize times every event resulting from the browser s interaction with the web application. Connection and authentication time is measured, all the way through Visual Completion, and each event is charted for easy visual inspection. Screenshot 3 shows the on-demand metrics for the page load in our case study.

6 Screenshot 3. Page Speed Details The Page Load Speed times each of these metrics: Domain Lookup Connect Secure Connection Request Response DomInteractive Dom Content Loaded Event Page Loading Time Dom Loading Dom Interactive Dom Content Loaded Event Visual Completion Another benefit of Functionize s platform and resultant performance data is that test cases can be selected to run by region from anywhere in the world. This allows for precise modeling of the user experience based on local regional performance. Regional granularity is an important performance metric feature to evaluate given the differences in cell network bandwidths, especially with mobile phones. We can disable AJAX cataloging, configure authentication settings, and many more advanced features to simulate every imaginable condition of operation for your app s performance. Functionize Provides Ultimate Performance As Agile and Devops have advanced in the areas of continuous integration, testing, and deployment, QA needs to keep pace. Common performance automation tools meant to achieve this equilibrium are limited due to the complexity of modern website applications and the requirement of scripting in most scenarios to model performance behavior. As traffic to sites and apps becomes increasingly difficult to predict for most enterprises, Functionize is building the next generation of performance benchmarking that makes it possible to understand, evaluate and model realistic simulations of users to your website.

Intelligent Mobile App Testing

White Paper Intelligent Mobile App Testing The field of software functional testing is undergoing a major transformation. What used to be an onerous manual process took a big step forward with the advent