Contents. Introduction Management issues. Technical issues. Dorothy Graham. Mark Fewster.

Transcription

1 1 Experience Driven Test Automation Mark Fewster Dorothy Graham Contents Introduction Management issues manager support setting appropriate goals success on a large scale Technical issues testware architecture life cycle for automated tests standards quality of scripts test effectiveness 2

2 2 New book: Experiences of Test Automation 29 case studies + 14 anecdotes collected, commented on and edited by Dot Graham and Mark Fewster to be published in 2011 by Addison Wesley contributions from a dozen countries applications from finance to space, IT to embedded both traditional and agile development Model-Based Testing, cloud, exploratory automation, tool-supported manual testing, test monkeys learn from others experiences 3 Contents Introduction Management issues manager support setting appropriate goals success on a large scale Technical issues testware architecture life cycle for automated tests standards quality of scripts test effectiveness 4

3 3 Manager support can make or break automation projects need to understand automation capability & limits building good automation takes time and effort and pays the greatest dividends building poor automation can be quick and cheap but costs outweigh benefits longer term need to manage automation projects not every test can or should be automated only automate good tests recognise automation and testing as different skills 5 Interesting manager stories the unbelievable case of propaganda already advertised 100% automated testing the mysterious case of death by stakeholders successful automation cancelled by stakeholders Think of it my way the classic case of inappropriate demarcation Bo Roop: automating testing of automation tool technical challenges particularly in maintenance manager: you write tests, others run & maintain but others couldn t maintain, tests abandoned 6

4 4 Interesting manager stories the unusual case of too many tools 1 st project success with non GUI application 2 nd project new tool, GUI application, no dev. co-op. 3 rd project different tool, framework/libraries 4 th project another tool, Python framework 5 th project new tool version, no to framework now automation deliverable for developers! the unsurprising case of spare time work 5 people part time no goals, no time to coordinate, re-invention, not publicised lasted 6 months 7 Setting appropriate goals the refreshing case of a clear goal problems after release, want improved reliability a few people, basic execution & check log automated tests found bugs but delayed the release to fix them management reaction? and another Lars Wahlberg: a senior manager set clear goals ( this helped ) provided adequate funds, monitored closely 50,000 new tests in 3 years, reusable for all projects 8

5 5 Automation objective: find lots of bugs? tests find bugs, not automation automation is a mechanism for running tests the bug-finding ability of a test is largely unaffected by the manner in which it is executed* this can be a dangerous objective particularly for regression automation! Manual Scripted Exploratory Testing Automated Tests Bug Fix Verification 507 (24%) 1229 (58%) 197 (9%) 178 (8%) Newman and Allen, CRM system 9 Success on a large scale ESA: Stefan Mohacsi, Armin Beer home-grown tool interfaced to commercial tools Model-Based Testing and Test Case Generation layers of abstraction for maintainability define model before software is ready capture and assign GUI objects later developers build in testability ROI calculations invest 460 hours in automation infrastructure break-even after 4 test cycles 10

6 6 Database testing: Henri van de Scheur tool developed in-house (now open source) agreed requirements up front 9 months, 4 developers in Java (right people) good architecture, started with quick wins flexible config., good reporting, metrics used to improve results: 2400 times more efficient from: 20 people, 40 tests, 6 platforms in 4 days to: 1 person, 200 tests, 10 platforms in 1 day quick dev tests, nightly regression, release tests life cycle of automated tests low maintenance, machine use 24x7, better quality 11 Contents Introduction Management issues manager support setting appropriate goals success on a large scale Technical issues testware architecture life cycle for automated tests standards quality of scripts test effectiveness 12

7 7 Testware architecture testware architecture (test framework) Testers write tests (in DSTL) High Level Keywords structured testware Structured Scripts Test Execution Tool runs scripts abstraction here = easier to write automated tests = widely used Test Automator(s) abstraction here = easier to change tools and maintain = long life 13 Implementing testware architecture tool will have its own way of organising tests generic solution may not suit will lock you in to that tool a better way (gives independence from tools) organise your tests to suit you, e.g. use pre-processing to copy files to where the tool expects to find them use post-processing to copy result files to where you want them to be it is about making the testers environment ergonomic to work efficiently with minimal mistakes 14

8 8 Designing testware architecture anecdote Molly Mahai, Test Manager set up good architecture? books not enough picked something to get started after a while, realised limitations too many projects, library cumbersome re-designed architecture, moved things around we didn t know what was needed until we experienced the problems for ourselves redesign quickly small impact learn on the fly versus pilot 15 Life cycle for automated tests case study Henri van de Scheur, testing databases, big success 5 year span, in-house tool, now open source < life cycle Suspend Development Candidate Active Retired Deleted different types of test suite MATS, Daily, Weekly, Non-Functional 16

9 9 Standards case studies Mecke, et al, major SAP application, >3 MLOC (auto) large government agency, 12+ automators, 100 s developers/testers coding standards readable and modular to encourage reuse and minimise maintenance effort document standards automated test specifications and reports process standards e.g. automated test development, execution, and maintenance 17 Quality of scripts case study and anecdote Mecke, et al, major SAP application, >3 MLOC (auto) Randy Rice, large organisation, big push to automate reviews review by automation expert useful (copied from dev.) daily technical reviews -> peer reviews -> forum static analysis on automation code in-house static analysis tool, routine checks e.g. reuse issues, attribute issues, naming conventions, zombie scripts continually gather new ideas for checks 18

10 10 Effectiveness of automated tests anecdotes Julian Harty various experience reports inherited API tests slow to run, days to update poor structure, only thing tested for: run-time exceptions, not does it do what it should deleted half, re-wrote rest based on unit tests networked tests all but one passed it was the only one that actually connected to the server the only one to indicate the truth! others said pass if there was no error message consider whether to improve or discard such tests 19 Summary: some key points management support and understanding is essential for long-term success appropriate objectives and resource, measure ROI a few technical issues good testware architecture helps life cycle for automated tests & standards quality of scripts & test effectiveness case studies provide many insights and ideas take advantage of them and improve your own chances of success 20