The Need for a Holistic Automation Solution to Overcome the Pitfalls in Test Automation

Transcription

1 The Need for a Holistic Automation Solution to Overcome the Pitfalls in Test Automation Abstract Many organizations opt for test automation with the primary objectives of reducing the cost of quality and improving the go- to market time. However, empirical data points out that majority of the test automation projects fail to yield the desired outcome and soon become abandoned. The chosen automation tool has a profound effect on the outcome of the engagement. This paper explores the tool and technology specific challenges and the desired characteristics of a good test automation tool. Copyright Virtusa Corporation. All Rights Reserved.

2 Introduction Present landscape The organizational spending on QA as a percentage of the total IT spending is increasing yearly. This rise in the QA portion of the IT budget is linked to the growing importance of application quality driven by digital transformation initiatives [1]. The increase in budget mandates QA departments prove their efficiency. Hence, organizations are increasingly trying to leverage automated testing with the objectives of improving test coverage, reducing test cycle duration, increasing the accuracy, and finding defects early in the life cycle, and reducing the cost of testing. However, research data shows that on average organizations have automated only around 28% of all test cases and organizations plan to reach 40% by 2017 [1]. Test Automation (TA) is no longer an option but a necessity for survival, particularly due to the increased adoption of digital transformation technologies, continued growth in the adoption of agile development Challenges in Test Automation Figure 1 - QA and Testing Share of IT Budgets. Adopted from [1] and methodologies and DevOps environments, the need for testing on multiple platforms largely due to the proliferation of mobile technologies. Test Automation tools have evolved over the decades from Text capture and Play back to modern day data driven and keyword driven frameworks [2]. However, the following perennial problems in these frameworks continue to discourage organizations from test automation. High initial cost - Test automation is an investment. This includes tool licensing and training. The free and Open Sources tools lack user -friendliness and completeness, thus many organizations opt for commercial tools. Automation of mobile applications requires significant additional investment on devices and Mobile Devices Management (MDM) services. Need for specialized resources with programming experience - Although the record and play back method is simple and can create automated test cases rapidly, its capabilities are severely inadequate for practical test automation. Thus, test engineers need to be conversant and experienced in scripting language used by the tool, as well as specialized in the automation tool. Framework development effort - Framework is the backbone of test automation. With no de-facto standards available, automation engineers tend to adopt their own designs and approaches. Poorly designed and poorly built frameworks result in high and often impossible maintenance efforts. Even with the readily available commercial or open source frameworks, it requires a significant amount of scripting (coding) to cater testing of the functionalities and custom controls of the application under test (AUT). High maintenance cost - Automated test suite becomes a code base that needs to be constantly updated to accommodate application changes. If the right level of standards and reuse is not applied, test scripts can become a hassle to maintain. Any initial benefit of Automation will be lost in the attempt to maintain the test scripts over a longer period. 2

3 Tool fragmentation - The modern organizations require testing their applications at different layers and paradigms. The applications mandate testing in the presentation layer on desktops and mobile devices as well as testing services and database layers. Achieving such a holistic and comprehensive automation with the aid of a single tool is nearly impossible. Thus, organizations are saddled with the burden of maintaining multiple code bases (frameworks and scripts) or compelled to reduce the automation scope to suit the capabilities of the chosen tool. Characteristics of Good Automation Solution The solution for viable and pragmatic test automation must have the following characteristics: Cost effective - Low initial investment on tool licensing, training and adoption cost. The commercial automation tools are quite costly particularly the ones that support mobile application automation. Supports end to end test automation - The tool should facilitate automation of the application on different layers (presentation, services and data) and on different paradigms (Web, Mobile, Desktop and Web Service). The tool should support a wide spectrum of development technologies. 3 Eliminates the need for programming skills - Organization should be able to use their domainsavvy manual testers for Test Automation rather than using programmers. Therefore, the tool should support easy learning and using without requiring programming knowledge. Accelerates automation and reduces maintenance effort - The framework development or customization time should be reduced to the core to facilitate faster automation of test cases to gain quick-wins. At the same time, the automation solution needs to be a robust solution that reduces the maintenance costs. Scalable - Supports scaling up (and down) as per the organizational needs. The tool should support rapid expansion of test automation, which is generally achieved through the ability to add execution nodes for parallel (grid) execution, support for an array of platforms (operating systems, browsers and devices), data and data model driven test execution, support for cloud and virtualization. Tool Selection An automation tool needs to be selected after carefully evaluating the multiple options and a proof of concept or through a pilot project. The evaluation criteria need to be defined for the short-term and the long-term Test Automation requirements, which should be in line with the broad QA roadmap of the organization. The following table provides a comparison of leading automation tools against a set of standard, high-level evaluation criteria

4 Criteria HP UFT IBM RFT Ease of Adoption Ease of Use Adoption Training - With Additional Costs - With Additional Costs Dedicated Support In-House Support Price (License + Maintenance) Open Source tool Stack (Selenium, Appium) High High Free Learning time Medium High High Scripting Language VB Script Java, C# Java, C#, +++ Scripting time Medium High High Object recognition Script Execution speed Required Programming knowledge Through Object Spy Through Object Spy High Low High Medium High High Selenium IDE, Fire- Bug, (3 rd party source viewers) Framework Needs to develop Needs to develop Needs to develop CI support Through 3 rd Party plugins Through 3 rd Party plugins Needs to build integrations Integration with Test Management tools With ALM With RQM 4 Flexibility and Extensibility Rich, Customizable Reports Execution Scheduling Execution Agents - Through 3 rd party solutions Mobile Web - Limited Mobile Native - Through 3 rd Party MDM or their MDM Supported Technologies Mobile Hybrid Web - Multi browser support - Through 3 rd Party MDM Desktop Applications Web Services Automation - Limited - With custom code - With custom code Table 1 - Comparison of leading automation tools against a set of high-level evaluation criteria. The data for this table was obtained from relevant user manuals, datasheets, and observations from the implementation of these tools in numerous real-life projects.

5 Conclusion It is evident that the test automation tools play a significant role in the outcome of an automation engagement. Therefore, selecting the right automation solution is of utmost importance just as setting correct objectives and realistic goals for test automation. Although there is large number of automation tools in the market, detailed evaluations reveal that the vast majority of these tools fail to fulfil the requirements of an enterprise in a holistic manner. Therefore, organizations opt to use multiple automation tools, perchance 3rd party add-ons, developing custom automation solutions, or invest in expensive third party automation service offerings. The need for a costeffective, comprehensive and scalable automation solution that can be used by non-technical testers remains unfulfilled References [1] Capgemini, Sogeti and HP, World Quality Report , [2] Linda Hayes, Evolution of Automated Software Testing - An Eye Witness Account, Automated Software Testing Magazine, August 2009 About Virtusa Virtusa is a leading worldwide provider of information technology (IT) consulting and outsourcing services. We help accelerate business outcomes for Global 2000 businesses in banking and financial services, insurance, healthcare, telecommunications and media. Virtusa delivers targeted business outcomes through differentiated solutions by leveraging its proven platforming methodology, domain expertise and technology innovation. Virtusa s operational excellence solutions allow businesses to improve business and operating efficiencies, and reduce costs using automation, effort compression and simplification. Virtusa s digital transformation and innovation solutions empower companies to accelerate business growth by reimagining the consumer experience, and creating lasting value. For more information write to: GTOSalesSupport@virtusa.com