CISC327 - So*ware Quality Assurance

Transcription

1 CISC327 - So*ware Quality Assurance Lecture 8 Introduc<on to Systema<c Tes<ng, part 1 CISC J.R. Cordy, S. Grant, J.S. Bradbury, J. Dunfield

2 Introduc<on to Systema<c Tes<ng Outline Today we begin a thorough look at so*ware tes<ng Defini<ons: What is so*ware tes<ng? Role of specifica<ons Levels of tes<ng: unit, integra<on, system, acceptance

3 What is Tes<ng? Tes<ng is the process of execu<ng so*ware in a controlled manner to answer a ques<on: "Does the so*ware behave as specified?" Implies that we have a specifica<on* Possibly that the tests are the specifica<on Or implies that we have some property we wish to test for independently of the specifica<on e.g., "All paths in the code are reachable, no dead code" Tes<ng is o*en associated with the words valida<on and verifica<on * compare h[p:// a- great- ar<st- in- just- 10- seconds- andi- mcclure- michael- brough/

4 What is Systema<c Tes<ng? An explicit discipline or procedure (a system) for choosing and crea<ng test cases execu<ng the tests and documen<ng the results evalua<ng the results, possibly automa<cally deciding when we are done (enough tes<ng)

5 What is Systema<c Tes<ng? Because in general it is impossible to ever test completely, each systema<c method chooses a par<cular point of view and tests only from that point of view (the test criterion) e.g., test only that every decision (if statement) can be executed either way

6 Verifica(on vs. Valida<on Verifica<on The checking or tes<ng of so*ware (or anything else) for conformance and consistency with a given specifica<on Answers the ques<on "are we doing the job right?" Tes<ng is most useful in verifica<on, although it is just one part of it Inspec<on, measurement, analysis and formal methods are also important

7 Verifica<on vs. Valida(on Valida<on The process of checking that what has been specified is what the user actually wanted Answers the ques<on "are we doing the right job?" Valida<on usually involves mee<ngs, reviews, and discussions to check that what has been specified is what was intended Tes<ng is less useful in valida<on, although it can have a role

8 Valida<on vs. Verifica<on Verifica<on Check that the so*ware meets its stated func<onal and non- func<onal requirements Valida<on More general than verifica<on, ensure that the so*ware meets the customer's expecta<ons Requirements specifica<ons do not always reflect the real wishes or needs of system customers and users

9 Tes<ng vs. Debugging Debugging is not Tes<ng Debugging is the process of analyzing and loca<ng bugs when the so*ware does not behave as expected Tes<ng plays the much more comprehensive role of methodically searching for and exposing bugs, not just fixing those that happen to show up by playing with the so*ware Debugging therefore supports tes<ng but cannot replace it However, no amount of tes<ng is guaranteed to find all bugs (except possibly exhaus<ve tes<ng, where prac<cal)

10 Exhaus(ve Tes(ng vs. Occasionally you can test exhaus<vely: Let s write all the tests for a NAND ( not AND or Sheffer stroke ) opera<on Tests are just the same as the specifica<on (truth table): A B A NAND B true true false true false true false true true false false true Only 4 possible combina<ons of A and B, so 4 test cases

11 Exhaus<ve Tes<ng vs. Most so*ware, even most individual methods, can t be exhaus<vely tested Examples: There are infinitely many integers Even a 32- bit integer (small these days) has ~4 billion possible values Strings: way too many strings to test Compilers are an extreme case: the set of possible inputs to a C compiler is the set of all C programs (and many almost C programs, e.g. programs with syntax errors)

12 The Role of Specifica<ons The Need for Specifica<on Valida<on and verifica<on ac<vi<es, such as tes<ng, cannot be meaningful unless we have a specifica<on for the so*ware The so*ware we are building could be a single module or class, or could be an en<re system Depending on the size of the project and the development methods, specifica<ons can range from a single page to a complex hierarchy of interrelated documents

13 Levels of Specifica<on Three Levels Specifica<ons of large systems usually contain at least three levels of so*ware specifica<on documents 1. Func<onal specifica<ons (or requirements) 2. Design specifica<ons 3. Detailed design specifica<ons

14 Levels of Specifica<on 1) Func<onal specifica<ons (requirements) Give a precise descrip<on of the required behaviour (func<onality) of the system Describe what the so*ware should do, not how it should do it May also describe constraints on how this can be achieved Example: When the user chooses the "Exit" menu item, bring up the "Save" dialog if the current document has not been saved, otherwise terminate the program

15 Levels of Specifica<on 2) Design specifica<ons Describe the architecture of the design to implement the func<onal specifica<on Describe the components of the so*ware and how they are to relate to one another Example: A UML diagram and associated documenta<on describing the rela<onship between a document object and a spelling checker

16 Levels of Specifica<on 3) Detailed design specifica<ons Describe how each component of the architecture, down to the individual code units, is to be implemented Example: A detailed descrip<on of the Document object, including the data structures used to store the informa<on, rela<onships with other objects, and so on

17 Levels of Tes<ng Corresponding Test Levels Given the hierarchy of specifica<ons, it is usual to structure tes<ng into three (or more) corresponding levels 3) (detailed design) Unit Tes<ng 2) (design specifica<ons) Integra(on Tes<ng 1) (func<onal specifica<ons) System Tes<ng To these levels, we usually add: 0) Acceptance Tes<ng

18 Levels of Tes<ng Corresponding Test Levels 3) Unit tes<ng addresses the verifica<on that individual components of the architecture meet their detailed design specifica<on 2) Integra(on tes<ng (a.k.a. component tes<ng) verifies that the groups of units corresponding to architectural elements of the design specifica<on can be integrated to work as a whole

19 Levels of Tes<ng Corresponding Test Levels 1) System tes<ng verifies that the integrated total product has the func<onality specified in the func<onal specifica<on 0) Acceptance tes<ng, in which the actual customers validate that the so*ware meets their real inten<ons as well as what has been func<onally specified, and accept the result

20 An Integral Task: Tests as Goals Once each level of specifica<on is wri[en, the next step is to write the tests for that level XP speeds this by making the tests themselves the specifica<on It is important that the tests be designed without knowledge of the so*ware implementa<on In XP, before implementa<on Otherwise we are tempted to simply test the so*ware for what it actually does, not what it should do

21 Using Tests Evalua<ng Tests Within each level of tes<ng, once the tests have been applied, test results are evaluated If a problem is encountered, then either: a) the tests are wrong: the tests are revised and applied again, or b) the soeware is wrong: the so*ware is fixed and the tests are applied again In either case, the tests are applied again, and so on, un<l no more problems are found, at which point development can proceed to the next level of tes<ng

22 Tests Don't Die! Test Evolu<on Tes<ng does not end when the so*ware is accepted by the customer Tests must be repeated, modified and extended to ensure that no exis<ng func<onality has been broken, and that any new func<onality is implemented according to the revised specifica<ons and design

23 Tests Don't Die! Test Evolu<on Maintenance of the tests for a system is a major part of the effort to maintain and evolve a so*ware system while retaining a high level of quality To make this con<nual tes<ng prac<cal, automa<on plays a large role in so*ware tes<ng methods

24 Summary Introduc<on to Tes<ng Tes<ng addresses primarily the verifica<on that so*ware meets its specifica<ons Without some kind of specifica<on, we cannot test Tes<ng is done at several levels, corresponding to the levels of func<onal, design, and detailed specifica<ons in reverse order Tes<ng is not finished at acceptance, it remains for the life of the so*ware system

25 Summary References Sommerville, ch. 8, "So*ware Tes<ng" The So*ware Test Page (on the web)