WHITEPAPER. Automated Functional Equivalence Testing. A White Paper by: Sagar M. Patil December, 2012

Size: px

Start display at page:

Download "WHITEPAPER. Automated Functional Equivalence Testing. A White Paper by: Sagar M. Patil December, 2012"

Jonah Fowler
6 years ago
Views:

1 WHITEPAPER Automated Functional Equivalence Testing A White Paper by: Sagar M. Patil December,

2 Table of Contents Executive Summary... 3 Typical Migration Testing Challenges... 4 Automation Approach for Functional Equivalence Testing Design Phase Script Development Phase Script Execution Phase Analysis... 6 Functional Equivalence Framework for any Migration Project... 7 Automation Framework for Enhanced ROI... 7 ROI Calculation... 8 Complete Application Migration is Planned Over Multiple Releases... 8 Conclusion About the Author About Datamatics Global Services

3 Executive Summary There is a growing need for migration from legacy Visual Basic 6.0 to VB.NET or Java technology as Microsoft has discontinued support for VB6. To leverage benefits of new technology and mitigate risk, organizations have started migration of their applications to.net or Java technology. For migration projects involving extensive testing, an automated functional equivalence testing solution is crucial Migration projects typically involve extensive testing. The key challenge is to ensure that functional equivalence is achieved on the target application. An Automated Functional Equivalence Testing (AFET) solution therefore becomes a crucial part of the migration process. This is reusable, smooth and cost effective means to achieve 100% functional Equivalence testing on any migration project. This Paper focuses on methodologies and techniques that Datamatics has developed for automation of functional testing, when a source VB project is migrated to.net. 3

4 Typical Migration Testing Challenges Datamatics AFET approach is to use single automation script on source and target platform When a legacy application is migrated as- is to a newer technology, e.g. VB (Source Application) to Java (Target Application) or VB to.net, some of the typical challenges faced during testing include: 1. Effort intensive Regression Testing 2. Time constraints for development and testing cycle 3. Manual testing - being labor intensive, contributes to higher costs 4. Insufficient test coverage 5. Data consistency verification 6. Lack of automation tools for functional equivalence testing Approach for AFET The functional equivalence tools available in the market support only bitmap comparison and have limited functionality when it comes to verification testing. Datamatics overcomes this constraint through use of a single automation script on source and target platform to achieve functional equivalence verification. Here is how it works: - The automation script is developed using QTP 11 [Functional Automation Testing Tool] - The business process transaction is scripted on the source platform and then executed on target platform - Any discrepancy from the source application is flagged as a failure on the target application execution - The verification includes comparison test both at controls (buttons, list box etc) and at functional level The high level steps in a testing process for AFET are: 1. Design 2.Script Development 3.Script Execution 4.Analysis Automated Functional Equivalence Testing Process 1 Design - Understand functionality - Analyze the reusable 2 Script Dev Develop description based script for reusable for VB as well as.net 3 Script Exec First execute script on VB application and mark results as baseline 4 Analysis Analyse the discrepancies against expected result and report errors Figure 3.0 4

5 1. Design Phase a. Source VB Application under Test (AUT) is analyzed for identification of reusable functions/actions that need creation of scripts b. Input and output parameters and dependences are documented c. Structure of Object Repository is analyzed QTP VB/.NET scripts The high level steps in a testing process for AFET are: 1. Design 2. Script Development 3. Script Execution 4. Analysis Microsoft Visual Basic6.0 VB Application Microsoft.NET.NET Application under Test Shared Framework Folder with reusable actions Script Execution Compare Results Report Errors Figure Script Development Phase a. Based on the analysis of reusable actions/functions, either objects from source VB application are added to object repository or a unique description of each object is created b. Descriptions are created in a manner such that each object is uniquely identifiable on VB (source) as well as.net (target) application c. Once the description of objects is created, object is added to object repository and functional logic is developed d. The process used to develop the Script ensures: i. Report displays total number of controls on form ii. The properties on VB and.net application (e.g visible\disable), total number of items in List box etc as well as excepted result of functionality are being tested e. It is important to note that while the scripts are being created on the VB application and the development of.net application is in progress, the developed scripts are fully compatible to run on the.net application f. Once.NET application is available, the automated scripts on the source (VB) application after minimal modification can be used for testing on target (.Net) Application 5

6 Shared Framework Folder with reusable actions Microsoft Visual Basic6.0 VB Application QTP Tool QTP Scripts Figure Script Execution Phase a. In script execution phase, automation scripts are executed on VB (Source) application and the results are stored in datasheets that can be later used as a baseline for test verification b. The same scripts with the data from stored datasheets are then executed on.net application followed by comparison of results against baseline c. Any deviance from the baseline is analyzed further Script Execution on Source Application Microsoft Visual Basic6.0 VB Application QTP Tool QTP VBApp Scripts Script Execution Base Line Results Script Execution on Destination Application Report Errors Microsoft.NET.NET Application Under Test QTP Tool QTP.NETApp Scripts Script Execution Compare Results Figure Analysis a. A Root cause analysis of scripts that failed on.net application is performed b. If script failure is due to functional issues, then it is reported as a defect and sent to the development team for fixes 6

Functional Equivalence Framework for any Migration Project With this approach, one can create a simple excel based framework for object identification properties and methods to be used by the tool

7 Functional Equivalence Framework for any Migration Project With this approach, one can create a simple excel based framework for object identification properties and methods to be used by the tool for object identification at runtime. Here is an example of object properties for VB and.net application A n automation framework helps achieve better productivity, faster turnaround time and higher accuracy Figure 4.0 Automation Framework for enhanced ROI ROI for AFET projects can be enhanced through use of frameworks. An automation framework helps achieve better productivity, faster turnaround time and higher accuracy. For the project being discussed in this paper, Datamatics proprietary Automation Regression Testing (DARTS) framework was used: DARTS Framework Key Benefits and features: 1. Complete control over the Test Planning, Execution and Test Execution Reporting Cycle 2. Technology Specific Component functions to access technology specific features 3. Greater control over AUT using built-in feature to check state of the object 7

8 4. Reusable Business actions built using component functions, aids to create granular test cases 5. Reusable architecture 6. Agile framework enables seamless data transfer across business scenarios and better control over execution for the tester 7. Recovery scenarios are integrated at the framework level. For example application types like web based, DotNet, VB based etc. ROI Calculation Better ROI on projects through AFET For calculating ROI on AFET projects, following factors need to be considered: 1. Cost per test automation resource 2. Number of test cases considered for automation 3. Automation tool cost 4. Time required for creating reusable scripts that work on VB as well as.net application 5. Releases planned per cycle 6. Number of regression cycles planned Complete Application Migration is Planned Over Multiple Releases Basis of ROI Calculation has been provided below Data Product Details Source Product Technology Migration Product Technology VB VB Vb.NET Vb.NET Testing Team Size Required For Current Scope Manual 4 Automation 2 Release Details No. of Releases Planned 4 8

9 Test Execution Data Data Source No. of Test Product CasesTechnology Migration No. of TestProduct Cycles Per Technology Release No. of Test Cases Increase Per Cycle 500 VB 2Vb.NET 50 Manual Testing Effort Parameters Total Test Cases Available 500 Cycle Duration in Working Days 10 No. of Working Days 5 Working Hours 8 Productivity 15 Automation Testing Effort Parameters Avg. Cost Manual/Automation Resource $40 US Per Hour No. of Execution Hours 8 Execution Productivity for 1 Cycle 30 Execution Productivity for 2 Cycle Onwards 45 Automation Tool License Cost $20000 US Figure 7.0 9

10 Note: From Release 4 onwards The Cumulative Cost of Automation Testing is lower than the cost of Only Manual Testing Figure 7.1 Cumulative Cost Graph Cost ($ US) Cycles Cumulative Manual Testing Total Cost Cumulative Automation Cost Figure

11 Conclusion Benefits of AFET: single script which is consistent, repeatable and accurate; reduced cost, time and efforts For projects involving migration of legacy applications to newer technologies, the discussed process for automated functional equivalence testing provide following benefits: 1. Single script can be used to test Functional Equivalence on Source and Target Application 2. Script execution is consistent, repeatable and accurate 3. Reduction in development and maintenance efforts for automation scripts. The automation strategy and frameworks can be used for any migration project 4. Reduced cost due to faster QA cycles and reduced manual testing efforts 5. This process enables the flexibility of choosing any automation tool for any migration testing project 11

About the Authors Sagar M. Patil Manager/Architect-Test Automation At Datamatics Sagar is responsible for providing automation solutions for diverse technologies and platforms.

12 About the Authors Sagar M. Patil Manager/Architect-Test Automation At Datamatics Sagar is responsible for providing automation solutions for diverse technologies and platforms. He has 12 years of experience in IT industry and his core expertise lies in Manual Testing, Test Automation, performance testing & Test Management. He has handled multiple complex testing projects using automation tools like QTP, Selenium, TestCompelte, CodedUI, Ranorex which includes framework development. Software Testing is his passion and he likes to address each challenge with an innovation but with simple solution. Sagar is PMP, CSTE, HP Functional test, Six sigma green belt certified professional. His white paper has been awarded in Testride Mumbai conference. About Datamatics Global Services Global Information Technology (IT) & Business Process Management (BPM) organization taking you forward on your route to Digital Transformation Trusted partner to several Fortune 500 companies globally Capabilities built around technology, domain expertise & knowledge of business processes Alliances with global technology leaders such as Microsoft, IBM & EMC2 More than 7,500 employees globally Certified for SEI CMMI Level 4 V1.3, ISO 27001:2013 & ISO 9001:2008 SSAE 16 compliant processes Global presence: U.S., UK, Australia, Dubai & India To know more, connect with us on business@datamatics.com 12

WHITEPAPER. Is Single Automation Framework possible for all Application Layers?

WHITEPAPER. Is Single Automation Framework possible for all Application Layers? WHITEPAPER Is Single Automation Framework possible for all Application Layers? A White Paper by: R. Ashok Kumar / Hitesh Malhotra / Naveen Agarwal / Ranjit Kumar September, 2013 1 Table of Contents Why