Tutorial to Building Automation Frameworksfor Web Services Testing

Transcription

1 STC 2013 Tutorial to Building Automation Frameworksfor Web Services Testing By: Tushar Sharad Archana Shenoy Deloitte Consulting India Ltd Plot No #14,15 RMZ Futura Deloitte Drive, Hitech City Hyderabad 1

2 Abstract Web service testing is used to test the functionality, load and scalability aspects of a web service. This tutorial takes a common scenario for testing Web services using Service Oriented Architecture and addresses the challenges faced. Some of the common challenges deal with a. the large number of regression tests b. testing effort required and man hours involved c. maintaining quality and consistency in manual testing of interfaces. We propose creating an automation framework using SOAP UI, which leverages manual test cases and makes it one of the most effective solutions.the proposed framework is designed for Web Services with XML/SOAP interfaces which provide testers with a means to detect and rectify common errors. A key element of this framework is the ability to perform root cause analysis of a failed test by comparing results with expected base responses. Additionally, the framework provides a mechanism to auto-correct such scenarios to reduce manual dependence and error impact. Introduction A web service is a software system designed to support machine-to-machine interaction over a network. Web services provide seamless connections from one software application to another over private intranets and the Internet. The major communication protocol used is SOAP being mainly XML over HTTP [1].Testing of web services is crucial to prevent late detection of errors, since it requires complex and costly repairs. There are a number of unique challenges to testing web services which can be overcome by the use of an automated testing tool. Web services-based architecture results in a distributed, loosely coupled system with dependent stakeholders. While this architecture holds great benefit in terms of scalability and extensibility of the system, it poses a great challenge for the tester. Another challenge in testing web services is that they are completely UI-less; they do not display a user interface that can be tested. This means that they hard to test manually, but are an ideal candidate for automated testing. Regression testing of web services is another area where automated testing is the preferred approach. Regression tests are normally a subset of a functional tests and the aim is to ensure that the web service is still working between builds or releases. The objective is to ensure that the area of functionality which was working in the previous builds / releases still does. Since regression testing is, by nature, a repetitive task, it is better if automated. While this applies to traditional applications and websites; it is even truer for UI-less web services. 2

3 Case Study Overview of Problem Testing of Web services has to target three aspects: the discovery of Web services, the data format exchanged, and request/response mechanisms While these three aspects provide and outline of our test plan;a major challenge for SOA test engineers is to test and perform vigorous testing to validate numerous web services without a user interface. Additional criteria such as test coverage and the number of negative scenarios help us identify the technical boundaries of our test framework. Test Coverage deals with ensuring adequate coverage of functional requirements, and Negative Testing focuses on testing for fault tolerance capability and recoverability of scenarios. In order to achieve this, we require a test harness and a test framework. Thorough functional coverage can be obtained by adding regression tests but it also poses additional challenges: 1. During SOA testing, time to delivery incurs large overheads due to rigorous testing of web services without the aid of user interfaces. 2. Modification of services late in the development life cycle requiresadditional testing effort. 3. Ensuring adequate functional coverage of requirements requires repeated testing. 4. Testers are responsible for isolating and reporting error scenarios which are time consuming activities. Since Web services are composed of loosely coupled interfaces distributed over networks, we must test the application: end to end service by service And interface by interface. To achieve a high degree of consistency, thoroughness and efficiency, a test framework must address these challenges as they have an adverse effect on testing timelines. Automation testing of web services significantly reduces time required for testing while maintaining the quality at the required standards. Solution The data format and request/response mechanisms can be tested by a single approach: invoke requests and observe responses with test data representing valid and invalid data formats. However, the dynamic features of Web Services impose numerous new challenges to traditional testing techniques. First, services are published, bound, invoked and integrated at runtime. The objective is to ensure adequate Regression Testing of web services. In order to incorporate efficiency into the framework, testing process needs 3

4 to be automated starting with test execution, test results collection, automatic test case correction, and analysis [1, 2]. The frameworkwas built primarily to provide a GUI for manual testing of web services and improve efficiency, however new functionality was added iteratively which led to the creation of an automation framework for testing web services. In each iteration/release, the focus was to create a framework based on the following principles 1. Data Driven Framework - This would enable users to test different business scenarios. 2. Batch Process Execution - This would reduce manual effort in the testing process. 3. Standard Reporting Structure - This would provide standardized reports for easy defect fixing. Improve test case execution process. Data Driven Framework Batch Process Execution Provide adequate functional coverage. Reporting Structure Maintain reporting standards. Ensure automatic test case correction. Regression Test Framework This frameworkcan be easily implemented and reused across multiple projects. The framework is built using the SOAPUI tool which leverages groovy scripting and ANT builds/compiles it. Fig1: Solution Framework Considerations 4

5 Framework Description Web Services testing activity in this scenario pertained to regression testing of Web Service Interfaces. This was achieved through three versions of the automation framework, of which the first version is used for basic testing of web services i.e. Basic, Alternative and Negative Scenarios. The framework incorporated the following functionality with each release: Version Functionality 1.0 a. Automation of manual testing process. b. Comparison of actual vs. expected outcomes. c. Summary reporting of test results 2.0 a. Test Data sheet contains Record Not Found correction queries. b. GUI interface 3.0 a. Test Data sheet contains Data Mismatch correction queries. b. Comparing responses by querying the database. Use a. Reduction of manual effort. b. Easy identification of error for defect logging. c. Adequate reporting made batch execution of test scenarios possible and effective a. Easy to use and simple GUI helped with the execution of Regression scenarios. a. The framework does a root cause analysis of the two most common error scenarios and auto corrects the faulty base responses if possible. Fig2: Framework Release Summary The figure below (refer Fig3) shows the first version of the automation framework. The process flow is as follows: 1. User creates test data with respect to different test cases/scenarios. 1.1 Tool picks up this data as input parameters and creates a SOAP request. 1.2 The SOAP request hits the database and fetches the corresponding response. 1.3 With the help of the actual results a high level scenario report is created. 1.4 The actual result is compared with expected results to create a data comparison report. 5

6 Fig3: Automation Framework Version 1.0 The second version of the automation framework has the added capability to handle Record Not Found flows. This situation occurs when test data specified in the test criteria is not found by the Web Service. This scenario requires a manual tester to check the web service and modify test data in compliance to the Database entries which exist. This version provides a GUI for execution of test scenarios. Fig4: GUI of the Automation Framework In the diagram shown below (refer Fig5); process 2.0 continues after step 1 has been completed and response is Data Not Found. The process flow is as follows: 2. Actual result for the test case is Data Not Found. 2.1 The tool updates the test case with the Status Flag as Fail. 2.2 For all those test cases where the status flag is fail, execute the corresponding database queries. 2.3 Update the test data with the results of the query. 2.4 Execute the test case once again using SOAP UI. 2.5 Update the expected base response for the failed Record Not Found test cases. 6

7 Fig5: Automation Framework Version 2.0 Version 3.0 handles all the scenarios mentioned in the previous versions and the Data Mismatch scenario. This scenario occurs when the test case has passed all Web Service checks and provides a result. However, this result is not the expected behavior of the test case. There are only two conclusions to such a response, either the database has been updated with a new record and thus the new result does not match our base response, or the web service is failing and does not perform as expected. The automation framework handles this scenario but is dependent on SQL queries to perform the auto-correct operations. It queries the database with the test data input and compares the result to the records contained in the base result. If they match, the web service is not performing as expected whereas if they do not match, it infers that the database has changed and proceeds to update the test case as well as the expected base result. In the diagram below (refer Fig6); process 3.0 continues after step 1 has been completed and actual response does not match the expected response. The process flow is as follows: 3. Data Mismatch 3.1 Find all test cases where data mismatch has occurred. 3.2 Execute corresponding database queries for each failed test case. 3.3 Compare the results of the query with the expected base results. 3.4 If they match, the interface has failed (since the test case failed yet the database contains the correct response); otherwise the database value has changed. Update the expected base response with the result of the database query. 7

8 Fig6:Automation Framework Version 3.0 Benefits and Limitations Benefits The framework greatly improves efficiency and the time savings are immense. Following are some quantitative benefits of the Web Service Automation framework adopted. 1. Reusability:The test cases can be used in various versions of the software. Not only will your project management stakeholders be very grateful for the reduced project time and cost, but it will certainly help you when estimating project costs. 2. Reliability: Automation of test script execution eliminates the possibility of human error when the same sequence of actions is repeated again and again. 3. Repeatable:We can test how the software reacts under repeated execution of the same operations. 4. Increased productivity due to automation of the testing process. 5. Increased turn out time for regression testing due to introduction of GUI in web services testing. 6. Increased quality of software components and application. Since the introduction of the automation framework more than 60% of total execution time was reduced, with an added reduction in analysis and design times as well. Limitations The automation framework is dependent on test data; which means that the User needs to create appropriate test data for each test scenario to be executed. This 8

9 involves building queries, especially for Versions 2.0 and 3.0 of the Framework. The framework would require further enhancement to facilitate the initial test data set up, query building and parameterization. A logical next step would be scaling it to a keyword driven framework, but such frameworks are complex, and would require a lot of groundwork in terms of planning, development of generic customizable and reusable functions. Rather changing it to a hybrid framework is a better alternative. It is action oriented and data driven and would set the baseline for converting it into a keyword driven framework if need be. Task for each Interface Time Taken per Interface (in Hrs.) Requirement Analysis 10 Identifying Test Scenarios 6 Creation of Test Case 14 Identify Test Data 15 Avg. time Consumed for manual execution of Test Cases Avg. time Consumed for automation execution of Test Cases Fig6: Metrics for savings per interface Execution Cycle Time Taken (in Hrs.) Time Saved (in Hrs.) Cycle 1 Execution NA (Manual) Cycle 2 Execution (Automation) Regression Regression Total Time Saved / % Fig7: Overall Time Savings 9

10 Conclusion In particular, test automation will be essential to a sound and efficient Web service development process, for theassessment of the functionality, performance and scalability of Web services as well as for theapproval and acceptance of Web services developed by application providers. This paper presents an automation framework which can be leveraged for rigorous testing of Web Services using SOAP UI. The framework is developed for Web Services with XML/SOAP interfaces and provides testers with a means to detect and rectify common errors. A key element of this framework is the ability to compare results with expected base responses. By determining the cause of failure of a test case, the framework provides a mechanism to auto-correct such scenarios thereby reducing the scope of error. References Automated Robustness Testing of Web Services - Evan Martin, Suranjana Basu, and Tao Xie Supporting Documents Details on customizing this framework are contained in the user guide below. Author Biography Automation Framework - User Gui Archana Shenoy has extensive experience on automation and worked on various tools such as QTP, RFT, VSTS and Selenium for Web Applications, Package Technologies, Web Services, Mainframe systems. She has also worked on building inhouse testing tools and accelerators to facilitate and overcome common testing challenges. She likes to work on packaging automation frameworks for maximum reusability. Tushar Sharad has over 2 years of experience working on automation using different tools such as QTP, RFT and SOAP UI. He has worked on Web Services, Web Applications and Mainframe systems. He has worked across multiple profiles as a developer and is an ISTQB certified tester. 10

11 Appendix Coverage: It is a measure used to describe the degree to which the source code of a program is tested by a particular test suite. A program with high code coverage has been more thoroughly tested and has a lower chance of containing software bugs than a program with low code coverage. HTTP: It is an application protocol for distributed, collaborative, hypermedia information systems. Interface: Itis the point of interaction in this case between two different systems or applications. QTP: It is a software which provides functional and regression test automation for software applications and environments. RFT:It is a tool for automated testing of software applications which allows users to create tests that mimic the actions and assessments of a human tester. SOAP:Is defined as Simple Object Access Protocol, is a protocol specification for exchanging structured information in the implementation of Web Services in computer networks. Web Service:It is a software function provided at a network address over the web. XML:It is a markup language that defines a set of rules for encoding documents in a format that is both human-readable and machine-readable. 11