CHAPTER 6. Slide 6.1 TESTING

Transcription

1 CHAPTER 6 Slide 6.1 TESTING

2 Overview Slide 6.2 Quality issues Nonexecution-based testing Execution-based testing What should be tested? Testing versus correctness proofs Who should perform execution-based testing? When testing stops

3 Testing Slide 6.3 Two types of testing Execution-based testing Nonexecution-based testing

4 Testing (contd) Slide 6.4 V & V Verification» Determine if the phase was completed correctly Validation» Determine if the product as a whole satisfies its requirements

5 Testing (contd) Slide 6.5 Warning Verify also used for all nonexecution-based testing

6 Software Quality Slide 6.6 Not excellence Extent to which software satisfies its specifications Software Quality Assurance (SQA) Goes far beyond V & V Managerial independence» development group» SQA group

7 Nonexecution-Based Testing Slide 6.7 Underlying principles We should not review our own work Group synergy

8 Walkthroughs Slide members, chaired by SQA Preparation lists of items Inspection Up to 2 hours Detect, don t correct Document-driven, not participant-driven Verbalization leads to fault finding Performance appraisal

9 Inspections Slide 6.9 Five-stage process Overview Preparation, aided by statistics of fault types Inspection Rework Follow-up

10 Fault statistics Slide 6.10 Recorded by severity and fault type Compare with previous products What if there are a disproportionate number of faults in a module? Carry forward fault statistics to the next phase

11 Statistics on Inspections Slide % of all detected faults (IBM, 1976) 70% of all detected faults (IBM, 1978) 93% of all detected faults (IBM, 1986) 90% decrease in cost of detecting fault (Switching system, 1986) 4 major faults, 14 minor faults per 2 hours (JPL, 1990). Savings of $25,000 per inspection Number of faults decreased exponentially by phase (JPL, 1992) Warning Fault statistics and performance appraisal

12 Metrics for Inspections Slide 6.12 Fault density (e.g., faults per KLOC) Fault detection rate (e.g., faults detected per hour) By severity (major/minor), by phase What does a 50% increase in the fault detection rate mean?

13 Execution-Based Testing Slide 6.13 Definitions Failure (incorrect behavior) Fault (NOT bug ) Error (mistake made by programmer) Nonsensical statement Testing is demonstration that faults are not present

14 What is execution-based tested? Slide 6.14 The process of inferring certain behavioral properties of product based, in part, on results of executing product in known environment with selected inputs. Inference Known environment Selected inputs But what should be tested?

15 Utility Slide 6.15 Does it meet user s needs? Ease of use Useful functions Cost-effectiveness

16 Reliability Slide 6.16 Frequency and criticality of failure Mean time between failures Mean time to repair Mean time, cost to repair results of failure

17 Robustness Slide 6.17 Range of operating conditions Possibility of unacceptable results with valid input Effect of invalid input

18 Performance Slide 6.18 Extent to which space and time constraints are met Real-time software

19 Correctness of specifications Slide 6.19 Incorrect specification for a sort Function tricksort which satisfies this specification:`

20 Correctness of specifications (cont d) Slide 6.20 correct specification for a sort:

21 Correctness Proofs Slide 6.21 Alternative to execution-based testing

22 Example of Correctness Proof Slide 6.22 Code segment to be proven correct

23 Example of Correctness Proof (contd) Slide 6.23 Flowchart of code segment

24 Example of Correctness Proof Slide 6.24

25 Slide 6.25 x y q, r q, r ) X=7, y=2 q=3, r=1 : x== y*q + r

26 Code 1 Slide 6.26

27 Code 2 Slide 6.27

28 Code 3 Slide 6.28

29 Code 4 Slide 6.29

30 Error code? Slide 6.30 what if x= 6, y=3? q=1, r=3

31 Code 5 Slide 6.31

32 Still error code? Slide 6.32 What if x is negative? r=-x, q=0

33 Code 6 Slide 6.33

34 Loop invariant Slide 6.34

35 Pre-, Post-condition Slide 6.35

36 JML example Slide 6.36 public class SqrtExample { public final static double eps = ; /*@ ensures x >= && JMLDouble.approximatelyEqualTo(x, \result * \result, /*@ signals (IllegalArgumentException e.getmessage()!= null &&!(x > public static double sqrt(double x) { if (x >= 0) { return internalsqrt(x); } else { throw new IllegalArgumentException("x is negative: " + x); } } //@ requires x >= 0.0; //@ ensures JMLDouble.approximatelyEqualTo(x, \result * \result, eps); protected static double internalsqrt(double x) { return Math.sqrt(x); } }

37 Slide 6.37 public class Person { private String name; private int weight; /*@ ensures \result!= && (* \result is a form of this person public String tostring() { return "Person(\"" + name + "\"," + weight ")"; } public void addkgs(int kgs) { if (kgs >= 0) { weight += kgs; } else { throw new IllegalArgumentException(); } } public Person(String n) { name = n; weight = 0; } } public int getweight() { return weight; }

38 Slide 6.38 package org.jmlspecs.samples.jmltutorial; refine "Person.java"; public class Person { private /*@ spec_public String name; private /*@ int weight; /*@ public && weight >= //@ also //@ ensures \result!= null; public String tostring(); //@ also //@ ensures \result == weight; public /*@ int getweight(); /*@ ensures kgs >= && weight == \old(weight + signals (Exception e) kgs < && e public void addkgs(int kgs); /*@ requires n!= null ensures && weight == public Person(String n); }

39 Correctness Proof Case Study Slide 6.39 Never prove a program correct without testing it as well

40 Episode 1 Slide Naur Paper Naur text-processing problem Given a text consisting of words separated by blank or by nl (new line) characters, convert it to line-by-line form in accordance with following rules: (1) line breaks must be made only where given text has blank or nl ; (2) each line is filled as far as possible, as long as (3) no line will contain more than maxpos characters Naur constructed a procedure (25 lines of Algol 60), and informally proved its correctness

41 Episode 2 Slide Reviewer in Computing Reviews The first word of the first line is preceded by blank unless the first word is exactly maxpos characters long

42 Episode 3 Slide London finds 3 more faults Including: The procedure does not terminate unless a word longer than maxpos characters is encountered

43 Episode 4 Slide Goodenough and Gerhart find three further faults Including: The last word will not be output unless it is followed by blank or nl

44 Proofs and Software Engineering Slide 6.44 Lesson: Even if product is proved correct, it must STILL be tested.

45 Three Myths Slide 6.45 Software engineers do not have enough math for proofs Proving is too expensive to be practical Proving is too hard

46 Proofs and Software Engineering (contd) Slide 6.46 Can we trust a theorem prover? How to find input output specifications, loop invariants What if the specifications are wrong? Can never be sure that specifications or a verification system are correct

47 Proofs and Software Engineering (contd) Slide 6.47 Correctness proofs are a vital software engineering tool, WHERE APPROPRIATE. If Human lives are at stake Indicated by cost/benefit analysis Risk of not proving is too great Also, informal proofs can improve the quality of the product Assert statement

48 Who Performs Execution-Based Testing? Slide 6.48 Testing is destructive A successful test finds a fault Solution 1. The programmer does informal testing 2. SQA does systematic testing 3. The programmer debugs the module All test cases must be Planned beforehand, including expected output Retained afterwards

49 When Testing Can Stop Slide 6.49 Only when the product has been irrevocably retired