Integrated Functional and Non -Functional Testing for Agile

Transcription

1 Integrated Functional and Non-Functional Testing for Agile P a g e 1 Integrated Functional and Non -Functional Testing for Agile STC 2013 Arush Gupta Umesh Kanade Harbinger Systems Pvt. Ltd 139, "Siddhant", Survey No. 97/6, Off. Paud Road, Kothrud, Pune , India

2 Integrated Functional and Non-Functional Testing for Agile P a g e 2 Abstract Agile model is based on iterative and incremental development, where requirements and solutions evolve throughout the development life cycle. The success lies in implementing user stories in time boxed sprints with optimal quality. Test Strategy implemented in such practices conforms to functional flows of the application. However, the non-functional requirements are often left for the later iterations. This is in contrast to the principle of early testing. Performance Engineering, Security Analysis and Test Automation are considered as separate areas and are traditionally carried out in an unsynchronized way. Due to lack of an integrated process common challenges and solutions are often not shared between test teams. This paper describes a Test Process Framework to integrate functional and non-functional requirements much earlier in scrum based agile projects. The framework utilizes the benefits of their co-relation and validates design decisions in iterative builds. The paper also presents a case study on implementing this framework, using open source tools primarily Jmeter, Selenium Web Driver and OWASP utilities. The data presented here, is from an ongoing project. Introduction Enterprise applications being developed today implement multiple technologies to achieve user requirements such as real time responses, secure handling of large data sets and rich user experience. There is a constant scope to target greater audience and multiple devices. With such trends, the importance of non-functional testing has greatly amplified along the functional objectives. Early Sprints miss Non-Functional Requirements The traditional agile context does not include stories to describe performance or security goals at module level. These requirements are often defined when major building blocks of the product are already designed and implemented. At this stage, the bug cost is very high and might create significant impact on product design and delivery schedules. Test automation, often starts from the scratch at this point to prepare build verification or regression tests. At this stage, the team has already performed frequent workflows, tedious tasks during the previous sprints.

3 Integrated Functional and Non-Functional Testing for Agile P a g e 3 The major reason behind this delay is the complexity in defining non-functional requirements at granular level. In the early iterations, application UI is also prone to frequent changes due to modifications in user requirements. Coordination Gap across Test Flavors Due to lack of an integrated process, these test activities are often performed in an unsynchronized way and in isolation with each other. Performance team is certainly not benefited by the automation scripts. Also, there is no platform for the performance team to share valuable inputs with the security or functional team. It is very important to consider the fact that security analysis and performance engineering should go hand in hand. Any additional security checks would affect system performance and vice versa. Further, the functional flows from the previous sprints should be validated in an optimal way such that there is minimal rework, and the functional team can focus on the new stories from the existing sprint. Thus, there is a definite need for a systematic approach, which can define functional and non-functional requirements much early in the development cycle. This approach should certainly utilize the benefits of their co-relation, and validate design decisions in iterative builds. Integrated Test Process Framework for Agile Model This Framework is designed to create a collaborative test environment in the existing Agile Model. It is based on agile principles and offers close coordination, parallel execution and high traceability for functional and non-functional requirements in agile projects. The design drives the test activities in parallel, such that one activity compliments the other. Functional tests provide valuable inputs for performance engineering and performance tests can result into valuable inputs for security testing or usability analysis. It closely monitors the scope of sharing solutions for common challenges across the test flavors. It also helps the designers to understand necessary actions for ensuring proper quality of the product throughout the iterative builds. Figure 1 below, depicts basic building blocks in this framework :

4 Integrated Functional and Non-Functional Testing for Agile P a g e 4 Figure 1: Integrated Test Process Framework for Agile Model The framework demands automation expert, performance engineer and security analyst to be on board in scrums right from the design level. The team constantly focuses on the below items: Proposed design, or Implementations for user stories Technologies and protocols being implemented Interaction between various application modules Understanding database and architectural design With the above information in place, test activities are planned with a defined scope and are integrated with the current sprints. Test Automation Strategy At this stage, the purpose of test automation is not to prepare highly configurable end to end automation framework but to develop an automation base. The primary focus is to

5 Integrated Functional and Non-Functional Testing for Agile P a g e 5 avoid any tedious or repetitive work flows being performed by the QA team. Various activities performed in this process are shown in Figure 2. Selecting the Right Tool Tool selection is based on factors such as support for various UI components, ease of implementation, report format and cost. We should also consider the possibilities to consume utility libraries which are already developed for common tasks, across the projects. It is certainly an added advantage, if we can integrate the scripts with other test management tools. Designing Module based Test Repository Test repository has a modular design. It is based on different states or state transitions of the application. The various classes in this repository are developed in parallel with user stories. Figure 2: Test Automation Strategy Designing Utility classes and Low Level Functions The utility classes are prepared to accommodate common actions in the application. This helps in code reusability across various functional flows. Implementing Object Repository An XML based object repository is prepared to store properties of UI elements present in the application. This is important to accommodate frequent UI changes. Any change in the properties is updated at one place and there is no need to recompile the code. Prioritize Work Flows Here is the catch. The functional flows identified for performance analysis are considered as good candidate for test automation. These are rated as high priority test cases and are implemented first. The scripts are prepared based on predefined set of test data and are

6 Integrated Functional and Non-Functional Testing for Agile P a g e 6 shared with the performance team. These tests can be easily executed to avoid repetitive work flows and verify application health at various user loads. Evolving the Test Base Automation base is constantly evolved along the user stories, to accommodate build verification and regression tests. This is achieved with minimal design level modifications. This helps the test team to verify functional flows in iterative builds, with minimal rework and further concentrate on the new stories. Integrating Performance Engineering The framework defines the following activities to be performed for effective integration of performance engineering during the application development: Activity 0 : Test Environment Setup Activity 1 : Identify User Activities Activity 2 : Selecting the Right Tool Activity 3 : Work Load Modeling Activity 4 : Knowledge Management Activity 5 : Test Execution and gathering Results Activity 6 : Result Analysis Activity 7 : Baseline the Readings Test Environment Setup Explore the possibilities to use existing set ups, resources for performance activities. Identify the physical architecture of the system and configure Web, Application and Database servers used by the application. Identify User Activities The activities are identified based on critical transactions and frequent navigation paths in the application. In early sprints, the user activities can be in the form of web service calls or Ajax requests. Since the complete business flow is under development, these requests are considered for initial analysis and can be targeted with an appropriate load pattern. Selecting the Right Tool The choice of performance tool is based on factors such as support for protocols, load scalability cost and other features to simulate network traffic. We should also consider the

7 Integrated Functional and Non-Functional Testing for Agile P a g e 7 possibilities to write our custom logic or execute external code within the performance tool. For example: Open source tools such as Jmeter offers an interface, to run java scripts based on Selenium Web driver. This helps us in reducing manual efforts, as the automation scripts can be easily integrated to verify health of the application during load test executions. Workload Modeling Workload modeling demands deep understanding of the application modules with respect to complete design of the system. In early sprints, the user activity may not match the complete business flow. Thus it is very important to precisely calculate user load patterns. Any hard calculations would raise false alarms and generate invalid results. To get started with our analysis, the work load model should focus on the below items: Identify frequent actions, navigation paths in the application based on the user requirements Model user actions (add think times, avoid redundant requests) Start with a small user load Monitor Response Patterns Increase the user load in next test cycles and observe the below parameters: Response time Throughput (requests served per second) CPU and Memory utilizations on the given servers Database Resources (memory, connection pool numbers, deadlocks) Page Errors Knowledge Management This is an interesting phase in the framework, which offers close coordination across the test flavors. The objective is to share relevant information and executables based on the user stories. The below items are critical to ensure valid results of the tests: Physical Architecture Details (server configurations, logs) Observations (server errors, exceptions, usability issues, functional inconsistencies) Database scripts to create and remove test data. Performance scripts for quick analysis on application traffic Common challenges and solutions

8 Integrated Functional and Non-Functional Testing for Agile P a g e 8 Test Execution and Gathering Results The performance tests are carried out with a specific objective in mind. Multiple test cycles are performed to uncover the bottlenecks. These tests can be broadly divided as: 1. Load Tests: The objective is to identify system response under the given user load. These tests are designed to target peak usage of the application. 2. Endurance Test : The tests are designed to validate the stability of system. The user load is maintained on the servers for prolonged period of time. During test executions, relevant server resources are monitored and recorded. The generated data is used for further analysis. Result Analysis The analysis activities should be goal oriented and should always proceed in an appropriate direction. Based on the nature of the project, important counters are identified that require close monitoring in each test cycle. For ex: In case of Node Js and web socket based web applications, we might want to monitor total socket count on the application server. However this counter may not be critical for a Dot Net based application that requires close monitoring of IIS worker process. With this knowledge, the weak areas that must be uncovered as a part of performance engineering activities during application development are listed below: Long user response time Long server response Memory leaks High CPU usage Request queue length Database deadlocks Http errors, erroneous data returned

9 Integrated Functional and Non-Functional Testing for Agile P a g e 9 Null pointers and Illegal state exceptions Baseline the Readings Critical findings, observations made during the performance activities are shared with design team and stakeholders. In the current iteration, the application is evaluated on the below parameters: Response times for the given actions Bottlenecks uncovered (functions, static classes, DB queries) CPU and Memory Utilizations Error logs and Exception details Usability concerns if any, at the given user load Capacity planning for Production environment Based on the above findings, appropriate fixes are provided by the Design team. This is a continuous process that offers close monitoring of existing implementations, fixes and new stories being developed. It also provides important pointers for capacity planning on Production Environment. Integrating Security Analysis In the traditional agile model, security testing is often left for the later stages and is typically carried out once the performance activities are completed. This is majorly due to strict time bound sprints and limited visibility on mitigating these threats. The analysis typically uses a proxy tool to record, and scan complete application traffic to highlight various vulnerabilities. The application UI is often crawled with these tools to expose various URL s and hidden parameters. The automated tools certainly help in exploring possible vulnerabilities. However, they might not be impressive in exploring business logic defects or identify unnecessary logs created in the database during user interactions. Based on the analysis performed by the security experts, these tools can be used to exploit vulnerabilities in an optimized way. Figure 3, shows various activities to integrate security testing during application development:

10 Integrated Functional and Non-Functional Testing for Agile P a g e 10 Gathering Information Figure 3: Integrating Security Analysis Gather all the possible information about the application and the infrastructure it resides on. Have a separate application instance created for security testing. Monitor the traffic and analyze various protocols being implemented. Analyze session management, passwords, cookie handling in the current build. Understand the business logic for each parameter in various requests and how various modules interact. At this point, the inputs from performance team would be handy as the performance scripts would give an immediate heads up on the application traffic and significance of each parameter. The automation scripts can also be shared to create the required data in the application and avoid frequent reworks. Identify Entry Points Entry points are defined as client side interfaces that are required to send user data to the server. The entry points can be in form of input fields, hidden parameters and cookie data. Designing Penetration Tests Security tests are prepared with a specific objective in mind. The aim is not to create high volume DOS attacks in the current iteration but to identify bugs in Logical Architecture, which covers presentation layer, business layer and the database layer. With the help of intercepting tools or plug-ins, the vulnerable requests are targeted for possible attacks. It is a good practice to validate the application against OWASP (Open Web Application Security Project) top 10 threats. Analyze Results and Share the Possible Exploits The results are closely analyzed to avoid any false positives from the penetration tests. The possible exploits are shared with the design team and typically includes the below items: Request details (Absolute path, parameters, payload method) Response Data (Http code, exceptions, db logs) Impact (Priority, loss to system)

11 Integrated Functional and Non-Functional Testing for Agile P a g e 11 Resolution (Mitigation strategy) Reproduction Steps (Complete scenario details) It is very important to share the above findings with the Performance team as the mitigation strategy might affect system performance. The changes can impact the performance base lines and are required to be evaluated appropriately. Case Study on Implementing Test Process Framework in Agile, using Open Source Tools The assignment is to develop an Online Trading System. With this system, the users can view various active sports events and further place Bids and Asks at real time. The terms Bid and Ask are used to buy and sell contracts during the event. To define the scope for the current iteration, there can be five parallel events and 1000 users are expected to trade at real time. Design team was formed to implement the above stories and followed the agile development model. QA team was also introduced to monitor the designs being proposed and implemented. Following observations were made at initial stages: Technologies Implemented: Grails, Node Js, Ajax, Mongo and Redis DB. Protocols :Https, ws(web socket) Physical Architecture Required: Grails server(apache/tomcat), Node server, Mongo DB and Redis DB Based on the above observations, following objectives were defined for functional and nonfunctional requirements: Validate Create Event and Create Order user stories. Observe system behavior and server resources when an order is created, with load up to 1000 simultaneous users. Observe how modules interact and explore possible security vulnerabilities while creating an Order. We selected Selenium Web driver with Junit to automate the user stories. On performance front, Jmeter with web socket plug-in was implemented. To perform load tests for Create Orders scenario, it was required to create a new event in the application before every test execution. Hence we selected Create Event scenario to be

12 Integrated Functional and Non-Functional Testing for Agile P a g e 12 automated first, and further call this script from the performance tool. Figure 4 below, shows the test automation design which was developed in parallel to the user stories, in the current iteration. Figure 5 below, depicts the integration of selenium scripts in Jmeter. Figure 4:Test Automation Design Figure 5: Integrate Automation Scripts with performance tools This reduced constant reworks by the performance team as the selenium script created new events in the application for placing the orders, and then the load tests were triggered. Also with this script in place, the scenario was easily validated in iterative builds and the functional team was able to focus on the new stories. The automation test repository was further evolved to accommodate Delete Event, Edit Event and Create Order stories such that the complete flow could be validated. Performance tests were designed to target peak usage for placing the orders in the application. The user load was to be scaled from 100 to 1000 users across the test cycles. For 100 users, the results were good. However, for 250 users test we observed high error count. To analyze the root cause, we monitored critical resources on the servers using Sysstat utility available on Linux OS. Looking at the Node server results, we observed a sudden drop in total socket count. To fix the issue, u-limit (user process resource limits) was tuned to 40k on the Node Server. With this change, we were able to scale to 1000 users and the other resources were also monitored. This configuration was further replicated on Production Environment. Figure 6 below, shows the peak socket count on node server before optimization and Figure 7, depicts the peak socket count after the tuning was performed. The graph presented here are for two consecutive test executions:

13 Integrated Functional and Non-Functional Testing for Agile P a g e 13 Figure 6: Total Socket Count Before Server Tuning Figure 7: Total Socket Count After Server Tuning Security Testing was also performed in parallel with the above test activities, for the targeted user stories. The application traffic was closely monitored using OWASP tools such ZAP and Paros proxy to point out the probable requests for exploiting the threats. The performance scripts were also considered to get a quick understanding of various parameters along with the routine analysis. It was observed that the web socket request to create an order did not have any unique parameter, and was vulnerable to Parameter Manipulation attack. With this threat, User A was able to create an order in the system pretending as User B. The finding was shared along with the mitigation strategy. Conclusions Here are some of the conclusions we arrived at, while following this process framework: With functional and non-functional requirements defined and included in the early sprints; performance bottlenecks, security vulnerabilities are uncovered at granular level. This helps in creating a healthy system throughout the sprints. With test automation being implemented at an early stage, functional flows from the previous iterations are validated with minimal efforts, and the team can constantly focus on new stories. By correlating the test activities; common challenges, solutions and valuable inputs are shared throughout the test process. Thus, creating a collaborative test environment. The process also helps the design team on mitigating various threats and optimizing performance breaches. The continuous evaluation process brings in a positive change in attitude of the team, towards meeting the quality standards. With this framework, the team achieved satisfactory results on the production environment as the application was constantly evaluated across the critical parameters.

14 Integrated Functional and Non-Functional Testing for Agile P a g e 14 References Performance Testing Guidance for Web Applications, Microsoft Patterns and Practices. OWASP Project Performance testing with Jmeter Selenium Web Driver Author s Biography Arush Gupta Arush Gupta is a Bachelor of Engineering in Electronics and telecommunication from MIT Academy of Engineering, Pune (Maharashtra).He has got over 4 years of experience in Software Testing. He specializes in designing Test Automation Frameworks for Web-based and Mobile applications. He has also been providing solutions for work load modeling and capacity planning for web applications. Arush is also an EC Council certified Ethical Hacker. Currently Arush works at Harbinger Systems as Senior Software Test Engineer and senior member of Advanced Testing Services group. Umesh Kanade Umesh Kanade is the general manager of technology solutions at Harbinger Systems. With more than 14 years experience, Umesh has been actively involved in designing technology solutions with innovation and passion for a variety of businesses. Umesh heads the proposal engineering and Advanced Testing Services Group. He has been instrumental in driving the research and development portfolios at Harbinger, and is part of designing and delivering enterprise systems with expertise on the latest mobile, cloud, and big-data technologies. Umesh holds a bachelor s degree in computer engineering.