1 Testware Architecture Tel: +44 1558 685180 Fax: +44 1558 685181 Prepared and presented by Mark Fewster Grove Consultants Llwyncynhwyra, Cwmdu Llandeilo, SA19 7EW, UK Email: mark@grove.co.uk www.grove.co.uk Testware Architecture Contents What is testware architecture? Benefits and characteristics Aspects of testware architecture How to achieve consistency
2 What is testware architecture? testware: the stuff of which tests are made! - data files, scripts, expected results, docs., etc. architecture: governs structure/arrangement - where stored and used - how grouped and referenced enterprise - company / organisation - bold undertaking! Testware examples Script (ascii) Input Data Script (binary) Client names Log Captured Screen Data Expected Screen Data Diffs Doc. Test Report Account types Client accounts Accounts Report Account details Expected Accounts Report
3 Terms - Testware artefacts Testware Test Materials Test Results Input Scripts Products By-Products Documentation Data Expected Outcome Actual Outcome Log Status Difference Report Is testware architecture important? testers work with testware - documents, data, scripts, expected results,... - run test cases, check results (repeatedly) - analyse failures, report faults (repeatedly) - write test logs, summary reports to work efficiently - easily find required artefacts - minimal effort to name, store and reference - choose correct versions and test sets first time
4 Testware Architecture Contents What is testware architecture? Benefits and characteristics Aspects of testware architecture How to achieve consistency Benefits of a single approach likely to be well thought out & documented - ad hoc approaches have serious limitations shorter learning curve - for new people, for inter-project transfers more reuse of testware - between projects and products better use of automation - automating more testing tasks, e.g. housework economies of scale
5 Achieving good testware architecture good for what? - testware architecture must support the objectives of testing confidence support regression testing go back to old versions of testware target faults support selective testing enable individual testers to use different subsets of tests efficiency maximise test coverage or thoroughness within resource Objectives for testware architecture durable testware - needs minimal maintenance reusable testware - can be re-used in different environments/projects - flexible, encouraging re-use reduce testing costs - increase efficiency (easy to access and work with) increase effectiveness - target faults, flexibility, traceability
6 Testware Architecture Contents What is testware architecture? Benefits and characteristics Aspects of testware architecture How to achieve consistency Aspects of testware architecture testware structure in-house tool support (utilities) commercial tool support configuration management (impact, interface) procedures and standards test definition others?
7 Testware structure issues - scale, re-use/sharing, multiple version, platform and environment independence logical structure - library, test suites, testware sets physical structure - folder/directory hierarchy - file grouping - naming conventions Test materials logical structure Baseline Test Suite Configuration Items Script Set Test Set Data Set Utility Set Testware Artefacts Scripts Input Documentation Expected Outcome Data Utilities
8 Example Test Suite Enhancement Test Suite d_scribbletypical v2 d_scribblevolume s_scribbledocument v3 t_scribblebreadth t_scribblecheck t_scribbleformat t_scribblelist v3 t_scribbleprint t_scribblesave t_scribbletextedit v2 u_generalcompare Testware Library d_scribbletypical d_scribbletypical d_scribblevolume s_logging s_scribbledocument s_scribbledocument s_scribbledocument s_scribblenavigate t_scribblebreadth t_scribblecheck t_scribbleformat t_scribblelist t_scribblelist t_scribblelist t_scribbleprint t_scribblesave t_scribbletextedit t_scribbletextedit u_scribblefilters u_generalcompare v2 v2 v3 v2 v3 v2 In-house tool support typically small and very task specific - assume knowledge of testware structure - a few hours work to develop - command files or Perl, Python, JavaScript, etc. - examples audit, search/traceability, house-work, comparison Toolsmith specific role - has programming skills works with testers to identify new opportunities
9 Commercial tool support can be a significant undertaking - not too soon: get basic structure in place first - separate projects: selection and implementation don t short-cut these variety of tool types - execution automation most challenging - lots of other opportunities e.g. requirements analysis, static analysis, coverage measurement, test management Configuration management (CM) all testware must be under CM - no buts, no maybes, no excuses interface with CM should satisfy both sides - additional tool support may be necessary to bridge differences (typically in-house tools) principles - must help, not hinder - ideally CM invisible to testers
10 Testware library and CM Testware library = CM? - obvious place for configuration management - multiple versions - control access (at least prevent overwriting) Test suite CM? - unnecessary control - tester s s own domain - single version Procedures and standards applicable to all other aspects - fundamental to long term success - essential (not merely a nice to have ) prepare for rapid up-take - once benefits become apparent, improvements will sell themselves standardise form and style of documents - printed, on-line, HTML, etc.
11 Procedures and standards how to - add new tests, find existing ones - notify changes - impact analysis for proposed software changes naming conventions - for scripts, tests, data files, etc. information - guidelines to regime Test definition self contained description of each test case - test case specification purpose, test conditions inputs, test data (prerequisites), expected results environment information (e.g. file X must exist) - test procedure specification procedural execution instructions also may include - test planning, test design, execution preparation information, test logs
12 Test definition purpose to define test cases! - a single and complete source of information - can be any form text file, spreadsheet, database, other must be easy to work with (perhaps with tools) consistent layout/content - permits tools to access the information - increasing efficiency A test case Information Test input and instructions Prerequisites Expected results Actual results Artefact Test procedure Database Database data Test file Text file Database report Text file Database report Location d:\u\mark\testing\myapptests.doc c:\program files\myapp\database f:\myapp\envs\multi6.dat d:\u\mark\testing\test3.dat d:\u\mark\testing\exp\clients.dat d:\u\mark\testing\exp\report3.txt d:\u\mark\testing\clients.dat d:\u\mark\testing\db_report.txt
13 Example (partial) test definition ; TESTNAME: Save03 ; PURPOSE: Exercise save options. Name and purpose ; TEST MATERIALS SCRIPT Save.scp DATA EditMe.dcm DATA NoRead.dcm DATA d_scribbletypical\data\countries.dcm Test materials: required before start of test execution ; PRE-PROCESSING CopyFile EditMe.dcm CopyFile NoRead.dcm CopyFile d_scribbletypical\data\countries.dcm House work tasks to be completed before execution ; POST-PROCESSING DeleteFile NoRead.dcm ; TEST RESULTS SCRIB_NOCMP Countries.dcm SCRIB_DOC EditMe.dcm House work tasks to be completed after execution Actual and expected results and how they are to be compared Testware Architecture Contents What is testware architecture? Benefits and characteristics Aspects of testware architecture How to achieve consistency
14 Way forward main obstacle is the effort involved - ideally, everyone working with testware would spend a proportion of their time in developing a common testware architecture - this is likely to be chaotic! small, dedicated team - working in parallel with testers - providing the effort to analyse, build and sell - for small organisations, a team of one? Resources typical roles within the team - Testware Architect has overall view & vision of common architecture - Toolsmith experience programmer/analyst - Test Expert having good understanding of the testing undertaken across different areas people skills will also be important
15 Objectives of team to facilitate testers to test better by - providing a more effective environment - that is consistent across all project test teams - and is applicable, flexible, and extensible not to perform or automate tests - except as a means of furthering true objectives will have to sell the benefits of each change - persuade people to change working practices - demonstrate value to them Testware Architecture Summary Testware is the stuff of testing - architecture is getting organised to do the job Significant benefits can be achieved - requires a well thought out organisation Testware architecture as many facets - artefact placement, tool support, procedures, CM & TD Achieving consistency - a separate responsibility but with collective co-operation operation