Non-GUI Test Automation Concepts and Case Studies in Maintainable Testing

Transcription

1 Concepts and Case Studies in Maintainable Testing

2 Introduction Test automation is great, except when it s not Various reasons lead to test automation failure, some arising from automation via a GUI You can run automated tests through other interfaces Command lines APIs Data layers Network services We ll look at two examples of sophisticated automated test platforms One used the data layer The other used the network services layer Each using a flexible and maintainable keyword driven architecture. GUI focused test automation: a wrong turn whose time has come and gone Copyright (c) RBCS Page 2

3 Test Automation Zombie Apocalypse False positives False negatives Flaky tests Unreliable or incomplete test oracles Changing GUIs that break tests Disproportionate test maintenance costs Commercial tool licensing costs Test automation failure is scary. Not this scary, maybe Copyright (c) RBCS (Film: Dawn of the Dead. GIF: giphy.com) Page 3

4 Test Automation Implementation Best practices involve modularized automation, especially using keyword-driven design Keywords usually are high-level business interactions with a system, covering a number of detailed interactions Business-oriented testers can write tests in human language (albeit concrete tests) rather than using the tool s programming language This greatly improves maintainability over time as well as usability of the framework by the testers Business process models, use cases, and other models can be used to create the keywords The modelling can be manual or supported by tools Copyright (c) RBCS Page 4

5 Building a Keyword System Technical testers/sdets create a complex system which is designed to be easy to use Typical components include Intelligent front-end (e.g., drop-down controls based on context) Each keyword represents programmed functionality Execution virtual machine Executes keywords one-by-one Built in logging, error handling Test suite execution selector Long-term success requires a method for business-oriented testers to request more keywords Copyright (c) RBCS Page 5

6 Keyword-driven Automated Tests Test cases built from sequences of keywords, instructions, test data, and expected results Advantages: Keywords defined by business experts Business-focused testers automate tests Tests are maintainable Test specifications are independent of implementation Technical testers/sdets implement keywords via executable test scripts The creation of the test scripts must be integrated into the lifecycle These scripts cause the tools to read and run the test cases Business-focused testers execute the tests and analyze anomalies False positive rates can be higher, and must be isolated through manually repeating the test Sometimes, technical testers must help to isolate and resolve false positives Copyright (c) RBCS Page 6

7 Case Study 1: Environment Size, Diversity System under test manages highly diverse systems Agents running on each system interfaced with central workstation Central workstation allows administration of accounts on all the systems with agents on the network Copyright (c) RBCS Page 7

8 Case Study 1: Tool Architecture Copyright (c) RBCS Page 8

9 Case Study 2: Environment, Data Diversity System under test provides data analytics and extract capabilities from a diverse set of data repositories Provides command line interface (CLI) access to the data layer through English-like non-procedural commands CLI is highly stable and accessible via open-source scripting languages? Copyright (c) RBCS Page 9

10 Case Study 2: Data Analytics Tool Scripts make the whole thing work Expected result location can vary, so search is DBspecific, then OS-specific, then generic Smart comparator (written in C for speed) allowed masking of changeable fields and record-order variations Copyright (c) RBCS Page 10

11 Conclusion Test automation efforts sometimes come to grief Trying to automate tests through the GUI has proven particularly challenging Keyword-driven test automation is a best practice, but it s not just for GUI testing Keyword-driven test automation can be done at service layers and data layers, often using open-source tools Time to defenestrate your automated tests! Looking to sunset your GUI-based automation? Maybe you should be Copyright (c) RBCS Page 11

12 Contact RBCS For over 20 years, RBCS has delivered consulting, training, and expert services to clients, helping them with software and hardware testing. Employing the industry s most experienced and recognized consultants, RBCS advises its clients, trains their employees, conducts product testing, builds and improves testing groups, and hires testing staff for hundreds of clients worldwide. Ranging from Fortune 20 companies to start-ups, RBCS clients save time and money through improved product development, decreased tech support calls, improved corporate reputation and more. To learn more about RBCS, visit. Address: RBCS, Inc Beck Road Bulverde, TX USA Phone: +1 (830) info@rbcs-us.com @LaikaTestDog LinkedIn: YouTube: Copyright (c) RBCS Page 12