Testing Tools. Software Testing and Verification. Lecture 14. Stephen M. Thebaut, Ph.D. Prepared by. University of Florida

Transcription

1 Testing Tools Software Testing and Verification Lecture 14 Prepared by Stephen M. Thebaut, Ph.D. University of Florida

2 Overview of Topics Why is test automation desirable? Types of test automation test design tools test implementation (including testing frameworks ) tools test execution and other run-time tools Sources for testing tools One other Testing Tool: Test Plans and Planning

3 Why is test automation desirable? A recent survey result reported in SoftwareTech News (a DoD publication) confirms that software testing activity consumes a significant fraction of the overall development lifecycle. October 2007 Vol. 10, Number 3

4 Why is test automation desirable? (cont d) In the same article, a study undertaken by the US Navy showed that substantial time savings can be achieved via automated testing.

5 Other benefits of automated testing Some types of testing are not feasible without automation, e.g.: load ( concurrency ) testing long term soak testing to detect memory leaks/performance degradation More efficient build verification ( smoke ) testing Stand-alone lights out testing Increased repeatability Testers can focus on more creative tasks Much higher levels of coverage

6 Test Design Tools

7 Test Design Tools Parameter value generators Boundary value, equivalence class, and special value generators Random data generation based on PCFGs (Probabilistic Context-Free Grammars) Fault tree coverage Cause-Effect analysis Database test data generators (for performance, load, and usability test) (cont d)

8 Test Design Tools (cont d) Statistical test generators (for reliability, availability, and performance testing) Structural test case design tools: utilize code instrumentation (see Run- Time Tools) source code specific may incorporate symbolic execution capabilities to aid in path sensitization

9 Test Implementation Tools

10 Test Implementation Tools Test Case Scripting Languages: test scripts automate the steps involved in carrying out tests suitable for tests that will be repeated (i.e., in support of regression testing) scripts are maintained in a change-controlled Test Database (cont d)

11 Test Implementation Tools (cont d) Test Harness (scaffolding) Generators: automates development of test drivers, stubs, and oracles ( assertions ) often coupled with a test scripting capability (= testing frameworks ) useful for module and integration test

12 Run-Time Tools

13 Run-Time Tools Code Instrumentation: inserts additional code to monitor cumulative logic/dataflow coverage, support trace and debug functions, and detect run-time errors. aids in designing test cases to meet white-box coverage goals or to debug code. alters execution performance and may mask or introduce timing errors. many commercial (source code specific) tools available. among the oldest of testing tool types. (cont d)

14 Let s Pause for a Moment Imagine that it s summertime and that a 3- day weekend is just starting Wouldn t it be great to just grab a fishin pole and head on out to the lake!

15 Let s Pause for a Moment (Take 2) There s NOTHING like Paris in the the spring!

16 Run-Time Tools (cont d) Data / Output Comparators used to automatically compare output or data files for discrepancies effective in dealing with the Paris in the the spring problem most are not smart enough to ignore don t care data fields. useful for regression testing when combined with a keystroke recorder and playback tool Also know as inattentional or perceptual blindness. (cont d)

17 Run-Time Tools (cont d) Keystroke Recorder and Playback Tools records and plays-back keystrokes, pointer movement, and mouse clicks useful for regression testing when combined with a data/output comparator

18 Sources for Tools An excellent source of information about testing tools is the WWW. Use your favorite search engine to find sites associated with software testing tools (about 94 million Google hits at last count). Several QA and testing related magazines, newsletters, and trade journals are available which publish tool reviews and comparisons. (cont d)

19 Sources for Tools (cont d) Relevant links can also be found at the websites of several QA and testing related organizations and societies, e.g., SEI IEEE ACM American Society for Quality Society for Software Quality Quality Assurance Institute European Organization for Quality (cont d)

20 Sources for Tools (cont d) Some useful testing tools-related links:

21 Test Plans and Planning Pezze and Young, Software Testing and Analysis, Wiley, 2008

22 Motivation Planning just as important in testing as in any other engineering activity. Allows for the effective use of limited resources and management of activities. In a nutshell, it involves a disciplined, timely consideration of: What needs to be done, When and How it will be done, and What resources (including time and effort) will be required.

23 The Master Test Plan Also called a Comprehensive Test Plan. Must be compatible with overall project plan. Developed early during project planning and requirements stages. Defines requirements for lower level plans: Unit Test Plan Component Test Plan Product Test Plan System Test Plan

24 Master Test Plan Check-List Defines test stages (e.g., unit, integration, system) schedules, and responsibilities. Establishes objectives, strategies, and procedures. Identifies required tools, facilities, and test libraries. Sets criteria for test completion and success. Defines the integration test strategy and identifies all scaffolding required with responsibilities and schedules for its production. (cont d)

25 Master Test Plan Check-List (cont d) Defines test coverage goals and methods for tracking progress. Establishes a regression test policy. Specifies required hardware and software, particularly non-standard or special purpose elements. Identifies estimated resource requirements including machine time, disk storage, physical / logical device access, etc. Identifies training required. Specifies test entry and exit criteria.

26 Typical Test Entry Criteria Test plan is complete and has been reviewed Design and implementation of test cases reviewed and all problems fixed Drivers, stubs, oracles in place Test environment (hardware and software) is ready

27 Typical Test Exit Criteria All problems discovered have been recorded and fixed All fixes have been re-tested Regression test(s) complete All test cases and associated documentation updated as necessary

28 IEEE Test Plan Template 1. Introduction and references 2. Test items (versions, media, references, etc.) 3. Features to be tested 4. Features NOT to be tested (and reasons) 5. Testing strategies and approach 6. Dependencies 7. Test case success/failure criteria 8. Pass/fail criteria for the complete test 9. Test entry/exit criteria (cont d)

29 IEEE Test Plan Template (cont d) 10. Test deliverables/status communication vehicles 11. Testing tasks 12. Hardware and software requirements 13. Problem determination and correction responsibilities 14. Staffing and training needs/assignments 15. Test schedules 16. Risks and contingencies 17. Approvals

30 Test Planning Summary Test planning allows for the effective use of limited resources and the effective management of test activities. A plan is only as good as the planning that goes into it!

31 Testing Tools Software Testing and Verification Lecture 14 Prepared by Stephen M. Thebaut, Ph.D. University of Florida

32 Addendum: What are PCFGs? Similar to a CFG, a probabilistic contextfree grammar G can be defined by a quintuple: G = ( M, T, R, S, P ) where M is the set of non-terminal symbols T is the set of terminal symbols R is the set of production rules S is the start symbol P is the set of probabilities on production rules

33 PCFGs (cont d) The probability of a derivation (parse) is the product of the probabilities of the productions used in that derivation. Production rule probabilities are informed by observing distributions on a training set of similar composition to the language to be modeled. Languages generated by context-free grammars are known as context-free languages (CFL).

34 PCFGs (cont d) CFLs have many applications in programming languages, in particular, most arithmetic expressions are generated by CFGs. As a recent application, CFGs are an essential part of the Extensible Markup Language (XML).