Contents. Management issues. Technical issues. Mark Fewster.

Transcription

1 1 Experience Driven Test Automation Mark Fewster Contents Management issues management support setting appropriate goals success on a large scale Technical issues testware architecture life cycle for automated tests standards quality of scripts test effectiveness 3

2 2 Management support can make or break automation projects need to understand automation capability & limits building good automation takes time and effort and pays the greatest dividends building poor automation can be quick and cheap but costs outweigh benefits longer term need to manage automation projects not every test can or should be automated only automate good tests recognise automation and testing as different skills 4 Interesting manager stories the unbelievable case of propaganda already advertised 100% automated testing doomed to fail? Think of it my way the classic case of inappropriate demarcation Bo Roop: automating testing of automation tool technical challenges particularly in maintenance manager: you write tests, others run & maintain but others couldn t maintain, tests abandoned 5

3 3 Interesting manager stories the unsurprising case of spare time work 5 people part time no goals, no time to coordinate, re-invention, not publicised lasted 6 months the mysterious case of death by stakeholders George Wilkinson: NHS Care Records System distributed automation team many challenges overcome just beginning to pay dividends cancelled by stakeholders 6 Setting appropriate goals the refreshing case of a clear goal problems after release, want improved reliability a few people, basic execution & check log automated tests found bugs but delayed the release to fix them management reaction? and another Lars Wahlberg: a senior manager set clear goals ( this helped ) provided adequate funds, monitored closely 50,000 new tests in 3 years, reusable for all projects 7

4 4 CRM system: background and objectives Newman and Allen, USA 5 MLOC, 20 testers (18 auto.), UI testing, TestPartner goal: automation of QA regression tests one of top 3 corporate initiatives for the year mandate to run regression tests: consumed the time automation to free up testers for other high value added testing (exploratory, performance, load) within 1 year: reduce manual regression test time from 10 to 5 weeks, and automated regression from 2 weeks to 24 hours generated excitement and motivation, could track and show progress 8 CRM system: challenges gaining developer support (testability issues) sold benefits of faster and more frequent feedback harmony between testers and automators merge roles seeking constant improvement frequent meetings to review what worked and what didn t show-and-tell training team ownership 9

5 5 CRM system: benefits achieved reduction in regression test time manual regression test time cut from 10 to 5 weeks automated regression cut from 2 weeks to 24 hours no. of customer reported quality issues reduced despite more features and customers 98% 94% others 92% 90% 88% increased DDP from 86% 84% 87% to 97% over 4 releases 82% run automated regression every night on latest build Automation 9% Manual scripted 24% Exploratory 58% 96% Fix verification 9% 10 Database testing Henri van de Scheur, Norway tool developed in-house now open source: 9 months, 4 developers in Java (right people) agreed requirements up front earlier time to market consistent repeatable testing reduce cost of testing run tests more often run tests unattended find regression bugs good architecture, started with quick wins flexible config., good reporting, metrics used to improve 11

6 6 Database testing: results and pitfalls results (after 3 years): 2,400 times more efficient from: 20 people, 40 tests, 6 platforms in 4 days to: 1 person, 200 tests, 10 platforms in 1 day quick dev. tests, nightly regression, release tests low maintenance, machine use 24x7 better quality pitfalls testing matured at expense of simple things focus on automation at expense of usability 12 Contents Management issues manager support setting appropriate goals success on a large scale Technical issues testware architecture life cycle for automated tests standards quality of scripts test effectiveness 13

7 7 Testware architecture testware architecture (test framework) Testers write tests (in DSTL) High Level Keywords structured testware Structured Scripts Test Execution Tool runs scripts abstraction here = easier to write automated tests = widely used Test Automator(s) abstraction here = easier to change tools and maintain = long life 14 Implementing testware architecture tool will have its own way of organising tests generic solution may not suit will lock you in to that tool a better way (gives independence from tools) organise your tests to suit you, e.g. use pre-processing to copy files to where the tool expects to find them use post-processing to copy result files to where you want them to be it is about making the testers environment ergonomic to work efficiently with minimal mistakes 15

8 8 Designing testware architecture anecdote Molly Mahai, Test Manager set up good architecture? books not enough picked something to get started after a while, realised limitations too many projects, library cumbersome re-designed architecture, moved things around we didn t know what was needed until we experienced the problems for ourselves redesign quickly small impact learn on the fly versus pilot 16 Life cycle for automated tests case study Henri van de Scheur, testing databases, big success life cycle assign different status to tests depending on situation Suspend Development Candidate Active Retired Deleted different types of test suite MATS, Daily, Weekly, Non-Functional 17

9 9 Standards case studies Mecke, et al, major SAP application, >3 MLOC (auto) large government agency, 12+ automators, 100 s developers/testers coding standards readable and modular to encourage reuse and minimise maintenance effort document standards automated test specifications and reports process standards e.g. automated test development, execution, and maintenance 18 Quality of scripts case study and anecdote Mecke, et al, major SAP application, >3 MLOC (auto) Randy Rice, large organisation, big push to automate reviews review by automation expert useful (copied from dev.) daily technical reviews -> peer reviews -> forum static analysis on automation code in-house static analysis tool, routine checks e.g. reuse issues, attribute issues, naming conventions, zombie scripts continually gather new ideas for checks 19

10 10 Effectiveness of automated tests anecdotes Julian Harty various experience reports inherited API tests slow to run, days to update poor structure, only thing tested for: run-time exceptions, not does it do what it should deleted half, re-wrote rest based on unit tests networked tests all but one passed it was the only one that actually connected to the server the only one to indicate the truth! others said pass if there was no error message consider whether to improve or discard such tests 20 Summary: some key points management issues management support and understanding is essential for long-term success set appropriate objectives and resource, measure benefits technical issues good testware architecture helps life cycle for automated tests & standards quality of scripts & test effectiveness case studies provide many insights and ideas take advantage of them and improve your own chances of success 21

11 11 New book: Experiences of Test Automation 28 case studies + 14 anecdotes collected, commented on and edited by Dot Graham and Mark Fewster to be published in late 2011 by Addison Wesley contributions from a dozen countries applications from finance to space, IT to embedded both traditional and agile development Model-Based Testing, cloud, exploratory automation, tool-supported manual testing, test monkeys learn from others experiences 22