WHITEPAPER. Is Single Automation Framework possible for all Application Layers?

Transcription

1 WHITEPAPER Is Single Automation Framework possible for all Application Layers? A White Paper by: R. Ashok Kumar / Hitesh Malhotra / Naveen Agarwal / Ranjit Kumar September,

2 Table of Contents Why UI testing alone is not enough?... 3 Why test layers?... 3 We'll end up with the same layer N times!... 3 We'll end up wasting time fixing bugs that would never impact our users... 3 In that case, Why test end-to-end?... 3 Why test both?... 4 Finally, what are the objectives?... 4 Typical Application Layers... 4 Automated Layer-wise Testing... 5 Automation Tools for every layer... 5 Presentation Layer... 5 Web Based Tools... 5 Windows Based Tools... 6 Business Layer... 7 Database Layer... 8 Beyond Paradigm Integrated Framework... 9 Presentation Layer The Core Framework would include: Conclusion About the Author About Datamatics Global Services

3 Why UI testing alone is not enough? Success of any GUI application depends on how it interacts with user through its user interface, how the user actions are performed to access application's features and whether the application responds in functionally correct manner. An application with incorrect behaviour or invalid user interaction can lead to problems. It may not always be feasible to identify an incorrectly developed Business Function or an incorrect data relationship in a pure GUI Test. These problems can potentially be checked for and brought under control by testing at the other layers of the application viz - the business application layer where the actual logic is built in or the data access layer where in the Database structures & there interactions are defined. We will now try to look at why it is important to test at the individual layer level. Why test layers? The typical arguments which are put against doing any testing beyond the UI testing: We ll end up with the same layer N times! If an application is architected with the standard principles, the application is expected to have a clearly layered architecture and in this scenario, it is expected that each layer plays the specific role for which it has been designed. If you target your test at each layer and either mock away or include the layers below it, then you can write tests for each layer targeting the specific behaviour of that layer and nothing else. We ll end up wasting time fixing bugs that would never impact our users Are you sure? How can you be certain that that particular code path will never be hit by customers using the product from a layer higher up or possibly by a change in a dependency further down the layer stack? Why would you knowingly write code that s full of bugs and rely on the fact that what s true at a specific point in time will always be true? Think about all the time you ve saved by testing in layers and finding bugs early instead of late in the development cycle when integration testing usually happens. Why not use a little of that time to fix this type of bug? In that case, Why test end-to-end? Testing layers doesn t give you an excuse to not test from an end-user/end-to-end perspective. It doesn t matter if all the parts of a car are tested independently and work fine if they don t fit together in the end. It s critical to understand how components work together in a system and validate that assumptions made about the interfaces between them align from both sides of the fit. 3

4 Why test both? We think of it this way: when building a house, you wouldn t just get an inspection after the whole house is finished just to discover that the electrical wiring was done incorrectly, would you? r would you only test the electrical wiring once before all the drywall was up and forego the final walkthrough. To really have confidence that your house is built correctly, you have inspections for each major sub-system along the way (framing, plumbing, electrical, insulation, drywall, etc.) and then you do a final walkthrough for an integration/end-to-end kind of test. By testing all the components/layers independently, you re finding bugs in those components more efficiently than waiting until the whole system is put together. By testing from an integration perspective, you re testing the coupling of those individual components and their interactions. In both cases, you re using the right tool for the right job. Finally, what are the objectives? ŸEarly detection of the defects ŸJIT feedback to technical team ŸReduce regression test cycle time enabling faster production release cycles ŸMinimise manual test efforts ŸClean builds Typical Application Layers Today, applications are built on n-tier architecture. A typical such architecture is depicted below: It is to be noted that the number of layers depends on how the has been architected into a solution system along with its associated interfaces. 4

5 Further explanation on the layer wise testing assumes the following layers to be present in the application and also that the application is based on Microsoft.NET: Ÿ User Interface Layer (Web / Windows or a mixture of both) Ÿ Web Services Layer Ÿ Business Layer ŸData Access Layer Automated Layer-wise Testing one tool can satisfy the requirement for all the layers and hence it becomes kind of mandatory to use a set of tools most suited for the application set with various factors taken into consideration. Having identified the need for testing every layer and that one tool will not suffice for all the layers, let us now look at a quick evaluation of the tools to finalize what tools we will be using for what purpose. Automation Tools for every layer Presentation Layer As a part of presentation layer testing, the requirement is to test both Windows and Web based applications. Web Based Tools Based on various discussions that were held, two key web based tools have been compared here for testing the web applications presentation layer. 5

6 Parameter CodedUI Selenium Keyword and Data-driven testing Framework Partial Concurrent execution support on Multi OS / browser combinations CodedUI uses Selenium as the backbone to run multi-browser tests. Concurrent execution feature needs to be extended Supports multiple browser compatibility. (Concurrent execution feature needs to be extended) Reporting Minimal Open source Advance recovery (can be extended) Supports Windows Applications Windows Based Tools On a similar note, since the requirement to meet Windows application testing had also been there, the following table provides a brief comparison of some of the tools chosen for that purpose: Parameter Coded UI QTP Test Complete Visual Studio integration Seamless continuous integration Cost Low High Medium Efforts Maintenance Framework for object creation & repository (Can be extended) Inbuilt Recovery Scenario (Can be extended) Support to third party / custom control Playback performance Good Poor Average 6

7 Business Layer Web Services Testing Tools There are a number of good tools that can be used for and have been adapted to test web services. The table below presents a few that were close fit to our specification: Parameter Spec Flow SOAPUI Web Services Studio Visual Studio integration Support Typed Dataset (Licensed version) Seamless continuous integration Cost Efforts Support to WSE Kerberos Maintenance Load/Performance Testing Open Source Business Layer Testing Tools Understanding the various options and choosing the tools best suited to testing the business layer can reduce complexity and increase efficiency. We tried to evaluate some tools which qualified on the parameters of our pre-requisites: Parameter MSTest SpecFlow Fitnesse Visual Studio integration Seamless continuous integration On demand execution Debugging capability 7

8 Database Layer A good automated testing tool actually understands the contents of the data files and iterates over the contents in the automated test. We have compared some of the tools which had been in line to our needs: Parameter SQL Developer DBFit Integration with Oracle Command line execution support Support from Oracle (Comes with Oracle license) Easy of scripting Advance Reporting capability Remote Debugging 8

9 Beyond Paradigm Integrated Framework The idea is to develop a framework which can integrate all the tools in one. Integrated Automation Framework Integration Engine Framework Components Data Mapping Method Dependency In-Built Recovery Test Mgmt Tool Plug-in data Management Scheduler Reports Module Code Coverage UI Layer Common Components Configuration Security Logging Execution Engine QTP Coded UI SOAPUI SpecFlow Sql Developer Selenium High Level Architecture The team started analyzing each layer separately to check what s best for the application. While making selection of the tool sets, the following key criteria were considered: ŸLeverage existing tools sets ŸAvoiding too much variety of automation tools and keeping future maintenance costs to minimum 9

10 Presentation Layer: Web Based: Initially, Selenium was thought to be the best option as it is open source. However since the team wanted to minimize the number of tools, the fact that Coded UI supports multibrowser compatibility tests using Selenium plug-in for Visual Studio in addition to the fact that the customer already had the necessary licenses for Coded UI, Datamatics recommended that CodedUI be the automation tool of choice. Windows Based: CodedUI is highly integrated with VS Studio and faster to execute and therefore was selected as the tool of choice. Business Layer: In addition to the Web services, we also had a Business Logic which needed to be tested. For Web services, based on the comparison in the evaluation; SpecFlow became the tool of choice. For the Business layer, in view of the fact that MSTest was being used as the Integration test tool to the development environment, MSTest was chosen as the tool of choice. Database Layer: The database layer gets tested through the various layer tests that are performed from the UI layer through to the Business Layer. However, as the team understood, there were a few stored procedures that were directly called by certain interfaces. In view of this and to avoid introducing additional software, SQL Developer was chosen as the tool. The Overall set of Automation Tools required looked as below: CodedUI Specflow MSTest SQL Developer CodedUI Presentation Layer WebServices Business Layer Database Web Applications 10

11 The Core Framework would include: Security Module: The security module of the framework, in addition to catering to the security requirements of the automation framework, will reference the existing libraries and Web Proxy libraries. Data Mapping: Each test case will be mapped to appropriate data conditions within the framework database with the test case ID potentially the key. In addition, a user interface will be provided to update the mapping for ongoing changes. Method Dependency: The module will be used to identify and map the dependency between the test cases and the various web methods. This will be further enhanced by providing a user interface to update the dependency for ongoing changes. Data Management: A separate seed database will be identified and used for automation testing for all layers. ExcelSheet and.csv data files to be used to inject test data. Dynamic data will be populated using SQL Scripts. Data files will also be used to pass data to other interfacing applications during execution. Code Coverage: This module will extract code coverage from the various tools that are being used within the automation framework and generate a report of the same. Inbuilt Recovery: Inbuilt Recovery functions to be built for 100% execution even if there are failures inbetween. Test Management Tools Plug-in: A Plug-in will be created to integrate with Test Management tools and report the execution results directly in those tools. Execution Engine: The execution engine will cater to automated triggering of the test automation scripts based on specific events and will also have the provision to manually trigger the execution of the automation scripts. Scheduler: A Scheduler is to be provided to sequence the automation test packs based on the requirements on different client machines. It will also help in parallel execution of the packs. 11

12 Reports Module: The reporting engine will have the functionality to publish and test result report for associated build to all concerned stakeholders in a dashboard format. An option to configure new reports would be provided at minimal changes. The reports will also have screenshots for easy failure identification. Integration Engine: In addition to integrating the tools identified and finalized during the foundation phase, Build Management Tools will be integrated as part of a common function that will be used in the automation scripts to invoke the smoke builds directly. 12

13 Conclusion The idea is to achieve a combined single framework which can integrate the best industry standard tools to achieve end-to-end automation. The wrapper concept can bring about a sea-change in the way frameworks are developed today. Built in a modular structure, the framework will facilitate. P One Integrated Framework for P Inbuilt Recovery for 100% All Automation Tasks Execution P Faster time to market P Configurable Reporting P Leverage Code Coverage statistics for improvements P End-to-End test case automation spanning across modules, interfaces and batches simulating real life end user actions P Auto Integration with Test Management Tools P Support automation at all layers P Integration with Build Release P Cross-leverage test cases Tools across modules P Scheduler for Parallel Execution & Nightly Schedules P Time & Cost Savings P Better Quality Copyrights 2012 Datamatics Global Services Ltd. All rights reserved. 13

14 About the Authors R. Ashok Kumar Practice Head Testing Services At Datamatics, Ashok is responsible for Testing Services across the organization to facilitate quality delivery to the customers. He has more than 24 years of IT industry experience and his core expertise lies in project / program management and in providing solutions on Transformation, Test Automation frameworks and has been actively providing solutions on Transformation and Test Automation frameworks for the past 10 years. He has handled very complex international and domestic projects on Transformation, Development and Test Automation frameworks. He holds a Master s degree in Management & a Bachelor s in Engineering. Prior to Datamatics he has worked at Nuclear Power Corporation as a senior scientific officer. Hitesh Malhotra Test Manager Hitesh is a test manager at DGSL with more than 11 years of experience in Testing Services. He holds a Master s degree in Management & Bachelor s degree in Engineering. Before joining Datamatics he has worked in various companies like Microsoft, Dell & Thomson Reuters. He has been a key driver in implementation of various propriety Automation Frameworks. Also, looks on to deliver Business Value out of Automation Testing. Copyrights 2012 Datamatics Global Services Ltd. All rights reserved. 14

15 About the Authors Naveen Agarwal Senior Manager A Senior Manager at Datamatics, Naveen has an experience of 12 plus years in development of applications on both Microsoft and Java Technologies and has extensive experience in architecting solutions on.net. Naveen holds a Bachelor s degree in Engineering and is a certified Scrum master and a PMP. Ranjit Kumar Ranjit holds a Bachelor s in Engineering and has extensive experience in architecting.net solutions with more than 9 years of experience. He has contributed significantly in converting the concept of this white paper into a reality by guiding the team in developing this integrated framework. He has to his credit architecting other test automation solutions like Batch automation test engine and Dartenium. About Datamatics Global Services Global Information Technology (IT) & Business Process Management (BPM) organization taking you forward on your route to Digital Transformation Trusted partner to several Fortune 500 companies globally Capabilities built around technology, domain expertise & knowledge of business processes Alliances with global technology leaders such as Microsoft, IBM & EMC2 More than 7,500 employees globally Certified for SEI CMMI Level 4 V1.3, ISO 27001:2013 & ISO 9001:2008 SSAE 16 compliant processes Global presence: U.S., UK, Australia, Dubai & India To know more, connect with us on business@datamatics.com Copyrights 2012 Datamatics Global Services Ltd. All rights reserved. 15