Agile Test Automation Framework - Overhauling the Challenges By Merral Crasto Test Lead, IBM India Pvt Ltd. Email: mecrasto@in.ibm.com - 1 -
Table of Contents Abstract...3 About IBM India Pvt Ltd...3 The Framework...5 Framework Functioning...5 Framework Elements...6 Steps to Automate a Scenario...8 Benefits of the Framework...9 Author Biography...10-2 -
Abstract Basic Challenges of agile test automation are 1. Continuous Development 2. Frequently changing requirements How Agile Test Automation can help? Test Automation can help organizations resolve the challenges associated with Agile testing. Test Automation ensures that the application is and continues to be in a good shape with each new sprint. The "Agile Test Automation Framework" developed can support and address the challenges that are faced by many projects.this framework is a robust framework which will simplify and speed up scripting by making use of re-usability and help in automating end to end scenarios with less effort. This is hybrid framework where the whole scripting process requires creation of functions, which represents a small section of the application under test and Other highlight of the framework is keywords can be added on the go as the new applications/flows gets added to the execution flow. About IBM India Pvt Ltd. IBM is a global technology and innovation company that stands for progress. With operations in over 170 countries, IBMers around the world invent and integrate hardware, software and services to help forward-thinking enterprises, institutions and people everywhere succeed in building a smarter planet. IBM has been present in India since 1992. The diversity and breadth of the entire IBM portfolio of research, consulting, solutions, services, systems and software, uniquely distinguishes IBM India from other companies in the industry. IBM India's solutions and services span all major industries including financial services, healthcare, government, automotive, telecommunications and education, among others. As a trusted partner with wide-ranging service capabilities, IBM helps clients transform and succeed in challenging circumstances. IBM has been expanding its footprint in India - and has a presence in over 200 cities and towns across the country - either directly or through its strong business partner network. IBM India has clearly established itself as one of the leaders in the Indian Information Technology (IT) Industry - and continues to transform itself to align with global markets and geographies to grow this leadership position. Widely recognized as an employer of choice, IBM holds numerous awards for its industry-leading employment practices and policies. - 3 -
Basic Challenges of Agile Test Automation Volume and Speed of Change: An Agile principle: Welcome changing requirements even late in cycle Changes in definition of product and its correct behavior Agile methodologies are more faster Inconsistent or Inadequate Unit Testing: Short execution periods on Agile sprints lead to buggy code on system Not all developers do unit testing Increased Regression Risk: Code that worked in previous sprint get churned by new features in subsequent sprint - 4 -
The Framework Framework Architecture & Elements: Software application undergoes rigorous functional tests, especially supported by automated testing frameworks. Automating these frameworks and maintaining quality software releases are critical to business performance. By implementing the appropriate testing framework, enterprise can significantly increase the speed and accuracy of the testing process, provide a higher return on investment from Software project and systematically minimize risk. The Test Automation Framework must be modular, easy to maintain, reusable, and leveraged across multiple projects to maximize the value of test Automation Suite Common automation considerations for a Test Automation Framework include: Maximizing test coverage Scale for future requirements Reliability and Consistency of result Duration of execution Identification of all regression testing defects End to end testing capability to cover business scenarios Centralized Location Object Repository Automation Tool Datasheet Business Flow Constant & Env file Function File Datasheet Test Data Fig1: Framework Architecture This framework (Fig1: Framework Architecture) ensures that the application is and continues to be in a good shape with each new sprint. The "Agile Test Automation Framework" developed can support and address the challenges that are faced by many projects. This framework is a robust framework which will simplify and speed up scripting by making use of re-usability and help in automating end to end scenarios with less effort. This is hybrid framework where the whole scripting process - 5 -
requires creation of functions, which represents a small section of the application under test and Other highlight of the framework is keywords can be added on the go as the new applications/flows gets added to the execution flow. Framework Elements Functions: The function library mainly consists of generic functions and very few module-specific functions. These are used to perform some basic tasks. As the application revolves around performing common tasks like entering values to the fields, comparing values, retrieving values from the fields, clicking on links, images, and buttons, displaying messages, retrieving row and column counts, and sorting columns, the generic functions can be reused extensively. Datasheet Data is the heart of any application. The success/failure of a Test Case is determined by the kind of data that is used for testing the application. The purpose of a Data Sheet is to have data that will find faults in the system. If such a volume needs to be accommodated, then the maintainability of such a data sheet may turn out to be very high. To minimize the same, the concept of single data sheet is used for each driver script. Multiple sheets within a single datasheet are used to support different languages. With this concept, all the input and output data are maintained as a single excel sheet. This gives the flexibility of having the cells mapped so that any change in one cell data, which is common across, will be reflected in the other cells also. The datasheet has various rows and columns. The columns represent the various input and output values (parameters). The datasheet provides both input and expected results verification. Object Repository The object repository is a collection of objects like links, buttons, tables, and fields. While executing a test condition, these objects are recognized by QTP and mapped to the objects in the application rendering the test as success/failure. However, the need to have the objects in the repository is eliminated. The object repository is shared and module-specific. Driver Script - 6 -
The Driver script is the brain of this Automation Framework. All the concepts discussed so far like keyword-driven and data-driven approaches, are actually implemented by the Driver script. It traverses through the business flow data and calls the respective business components one by one. It takes care of executing the test case across all the test data iterations specified by the user. It is also responsible for generating the test reports at the end of test execution. It is a generic script, which can be used across all projects with minimum changes. Customized Test Results One of the key differentiators in this Automation Framework is the provision to provide customized test reports, over and above the results generated by the automation tool. The customized results are available in Excel and HTML formats, and are generated with the help of the support libraries. Constants & Environment Setting Files Environment File written in XML format. Depending on the environment either Production or Dev, we need to load the particular XML file. The settings that would go into these files are URL, DSN name, Login details for different environments, etc. Constant file has the parameters like frame name, object properties (link, button type, field type) that do not change frequently are captured in a central file. In the event of any change to the object properties, only the central file needs to be updated. This approach eliminates the need to make the same changes in multiple places thus ensuring maintainability. The changes made to the Constants file must be reflected in the Automation scripts to ensure that the scripts can be successfully executed. - 7 -
Steps to Automate a Scenario: Fig 2: Steps to Automate a Scenario The Driver Script is the entry point of the execution The Driver script reads the batch configuration - 8 -
The Driver script passes control to the relevant Business Components as specified by the Business Flow of the test case The Driver script and the components leverage the Reusable Libraries as required The components consume the Test Data specified in the datasheets The Driver script publishes the Test Results for all the test cases executed Benefits of the Framework Framework Features Reusable: Common utility functions are developed in such a way that the most of the functions are common across the application. The same function can be reused across the different application, versions of application and Modules. Independent: Framework enables to run test case by itself or in a suite with an arbitrary set of other tests in any order. Flexible: Framework is flexible to accommodate any enhancements without impacting existing functionality Minimal changes required for any object changes: Framework enables to patch any changes made to object easily without impacting the existing functionality. Supports Different version of application Framework uses external XML file to support different version of application and environment. Repeatable: Tests be run many times in a row without human intervention Maintainable: Less maintenance required because of common utilities properly put in place in this Framework. Due to this we have to change only the code of utilities and less time would be needed to change in actual scripts. Resulting in saving of time. Uniformity: With this framework, a uniformity will be attained in results, execution logs etc. This is by use of defined formats and reporting function in the scripts. Saves Time and Efforts: The framework works towards faster execution and easier maintenance. This would save execution time and also lesser effort to incorporate changes for new functionalities. - 9 -
Increase in Test case Coverage during Regression testing phase: Enhanced test coverage is possible with the framework. Owing to faster execution more exhaustive testing can be done Advantages and Benefits High degree of reuse within the application Object Properties can be changed without changing the script Flow can be changed without affecting the Object Properties Scripts run significantly faster than human being which in turn reduce the time spent on manual actions and test execution Scripts Precisely performs same operation each time eliminating human errors which ensures consistency Increase in Test case Coverage during Regression testing phase Generates Log File and Summary Reports Generates Email Reports to stake holders Supports Multiple Environments (SIT, UAT, Production) Easy to trace out the errors Informs the status of orders in each system for the Provided Work Flow Author Biography Merral Crasto joined IBM in 2006 and he has 7+ years of experience in Software Testing. He is CSTE and ISTQB certified Test Professional. He is a Test Lead in IBM. He has worked in ERP, Telecom, Retail domains. He has experience in both Manual and Automated Testing. - 10 -