MTAT : Software Testing

Transcription

1 MTAT : Software Testing Lecture 04: Static Testing (Inspection) and Defect Estimation (Textbook Ch. 10 & 12) Spring 2013 Dietmar Pfahl

2 Lecture Reading Chapter 10: Reviews (Lab 4) Types of reviews Defect estimation (not in textbook) Chapter 12: Evaluating Software Quality (no Lab) Usage-based testing Certification testing (not in textbook)

3 Structure of Lecture 4 Types of reviews Defect estimation Usage-based testing Certification testing

4 Reviews (Ch 10) Terminology Static testing testing without software execution Review meeting to evaluate software artifact Inspection formally defined review Walkthrough author guided review

5 Why Review? Main objective Detect faults Other objectives Inform Educate Learn from (other s) mistakes Improve! (Undetected) faults may affect software quality negatively during all steps of the development process!

6 Relative Cost of Faults Maintenance 200 Source: Davis, A.M., Software Requirements: analysis and specification (1990)

7 Reviews complement testing

8 Walkthrough Author guides through artifact ( static simulation ) Attendees scrutinize and question If defects are detected it s left to the author to correct them

9 Walkthrough Objective Detect faults Become familiar with the product Elements Planned meeting Team (2 to 7 people) Brainstorm Roles Presenter (author) Reviewers (Inspectors) Disadvantage Finds fewer faults than (formal) inspections

10 Inspections Objective: Detect faults Collect data Communicate information Roles Moderator Reviewers (Inspectors) Presenter Author Elements Formal process Planned, structured meeting Preparation important Team (3 to 6 people) Disadvantages Short-term cost

11 Inspection Process Fig 10.2

12 Defect Causal Analysis (DCA) Organizational Processes define Software Constr. (Analyse / Design / Code / Rework) Artifact Defect (Fault) Detection (Review / Test) prioritize & implement actions Action Team Meeting fix defects propose actions Defect find defects Database extract sample of defects Causal Analysis Meeting

13 Getting the best from reviews The author is in the hot seat How do you react? The development team Better prepared Feedback Communication The review team Critical thinking Ability to detect omissions Who should participate in the review? Cost-effective verification Minimising cost of correction Is it cost-effective?

14 Review Metrics Basic Size of review items Review time & effort Number of defects found Number of slipping defects found later Derived Defects found per review time or effort Defects found per artifact size Size per time or effort

15 Empirical Results Source: Runeson, P.; Andersson, C.; Thelin, T.; Andrews, A.; Berling, T.;, "What do we know about defect detection methods?, IEEE Software, vol.23, no.3, pp , May-June 2006

16 Inspections Empirical Results Requirements defects reviews good since finding defects early is cheaper Design defects inspections are both more efficient and more effective than testing Code defects - functional or structural testing is more effective and efficient than inspection. May be complementary regarding types of faults Generally, reported effectiveness is low Inspections find 25-50% of an artifact s defects Testing finds 30-60% of defects in the code

17 Reading Techniques Ad hoc Checklist-based Defect-based Scenario-based Usage-based Perspective-based

18 Perspective-based Reading User Designer Tester Scenarios Purpose Decrease overlap (redundancy) Improve effectiveness

19 Structure of Lecture 4 Types of reviews Defect estimation Usage-based testing Certification testing

20 Capture-Recapture Defect Estimation

21 Capture-Recapture Defect Estimation

22 Capture-Recapture Defect Estimation Situation: Two inspectors are assigned to inspect the same product (Lincoln-Petersen Model) d 1 : defects detected by Inspector 1 d 2 : defects detected by Inspector 2 d 12 : defects detected by Inspector 1 and Inspector 2 N t : total defects (detected and undetected) N r : remaining defects (undetected) N t d 1 d d 12 2 N r N t ( d d d )

23 Capture-Recapture Example Situation: Two inspectors are assigned to inspect the same product d 1 : 50 defects detected by Inspector 1 d 2 : 40 defects detected by Inspector 2 d 12 : 20 defects by both inspectors N t : total defects (detected and undetected) N r : remaining defects (undetected) d d N t N 100 ( ) 30 d r 12

24 Advanced Capture-Recapture Models Four basic models used for inspections Degree of freedom Prerequisites for all models All reviewers work independently of each other It is not allowed to inject or remove faults during inspection

25 Advanced Capture-Recapture Models Model Probability of defect being found is equal across... Defect Reviewer Estimator M0 Yes Yes Maximum-likelihood Mt Yes No Maximum-likelihood Chao s estimator Mh No Yes Jackknife Chao s estimator Mth No No Chao s estimator

26 Mt Model Maximum-likelihood: Mt = total marked animals (=faults) at the start of the t'th sampling interval Ct = total number of individuals sampled during interval t Rt = number of recaptures in the sample Ct An approximation of the maximum likelihood estimate of population size (N) is: SUM(Ct*Mt)/SUM(Rt) First resampling: M1=50 (first inspector) C1=40 (second inspector) R1=20 N=40*50/20=100 Second resampling: M2=70 (first and second inspector) C2=40 (third inspector) R2=30 N=(40*50+30*70)/(20+30)=4100/50=82 Third resampling: M3=80 C3=30 (fourth inspector) R3=30 N=( *80)/( )=6500/80= 81.xxx

27 Structure of Lecture 4 Types of reviews Defect estimation Usage-based testing Certification testing Software reliability

28 Software Quality (Chapter 12) 1. Quality relates to the degree to which a system, system component, or process meets specified requirements. 2. Quality relates to the degree to which a system, system component, or process meets customer, or user, needs or expectations.

29 Quality Attributes ISO 9126

30 Reliability Terminology Reliability: The probability that a system or a capability of a system functions without failure for a specified time in a specified environment Reliability Engineering: The discipline of ensuring that a system will be reliable when operated in a specified manner Reliability Engineering Goal: Developing software to reach the market within planned development time within planned development budget with known reliability

31 Statistical Testing NOT the same as ad-hoc testing! Sampling of tests (test data) follows a probability distribution Uniform (Random): probability of available candidate tests (test data) is equal Usage-based (Operational): probability of available candidate test (test data) follows an operational profile (i.e., a specific usage pattern)

32 Usage-based Testing Usage specification Test case generation Test execution Failure logging Certification, Reliability estimation Test Case Setup Call Failure Report #13 Output failure

33 Usage Specification Models Operational Profile State-Transition Diagram

34 Operational Profile Steps to develop an operational profile (Musa 1993) Definitions: 1. An operational profile is a quantitative characterization of how a software system will be used in its intended environment. 2. An operational profile is a specification of classes of inputs and the probability of their occurrence.

35 Operational Profile Customers Customer: person, group, or institution that is acquiring the software being developed. Customer Group: the set of customers that will be using the software in the same way. Customer Profile: the complete set of customer groups and their associated occurrence probabilities.

36 Operational Profile Users User: an individual, group or institution that actually uses a given software system. User Group: set of users who will engage the system in the same way. User Profile: set of user groups and their occurrence probability. Note: There might be different user groups for different customer groups

37 Operational Profile System Modes System Mode: a set of functions or operations grouped for convenience in order to analyze execution behavior. System Mode Profile: set of system modes and their occurrence probability. Example 1: administrator mode versus end-user mode Example 2: system usage during peak time vs. off-peak time

38 Operational Profile Functions Function: derived from system requirements, e.g., use cases Functional Profile: set of functions and their occurrence probability.

39 Operational Profile Operations Operation: are more specific than functions; they represent a specific task, with specific input variable values or ranges of values. In general, there may be more operations than functions associated with a system. Example: a function to modify a record could evolve into two operations: (i) delete old record (ii) add new record.

40 Operational Profile Example /1 The table shows an example operational profile of an ATM system (occurrences per day) Operation Occurrence Rate Occurrence Prob. Enter card Verify PIN Withdraw checking Withdraw savings Deposit checking Deposit savings Query status Test terminal Input to stolen card list Backup files Total

41 Operational Profile Example /2 The table shows an example operational profile of a component in a telephone system that is dedicated to forward incoming telephone calls to a certain telephone number at a certain point in time [Mus98]. The example profile provides a list of operations initiated by telephone subscribers, system administrators, the telephone network (external system), and the system controller (part of the system but external to the component). Operation initiator Operation Occurrence rate (per h) Subscriber Phone number entry 10, System administrator Telephone network Occurrence probability Add subscriber Delete subscriber Process voice call, no pager, answer 18, Process voice call, no pager, no answer 17, Process voice call, pager, answer 17, Process voice call, pager, answer on page Process voice call, pager, no answer on page 12, , Process fax call 15, System controller Audit section of phone number database Recover from hardware failure

42 Operational Profile Guiding Test Case Allocation Determine the threshold occurrence probability = 0.5 / #test_cases. Assign one test case to each infrequent operation. 3 1 Operations Infrequent Critical Assign the remaining test cases to the remaining operations in accordance with the occurrence probabilities. 2 Identify rarely occurring critical operations and assign 2-4 test cases to each.

43 Allocating Test Cases Example /1 Total number of test cases: 500 Threshold occurrence probability: 0.5 / 500 = Suppose that the number of infrequent operations with occurrence probabilities below threshold is 2. Assign 1 test case to each infrequent operation. 2. Suppose that we have one critical operation. Assign 2 test cases to it. 3. Distribute the remaining (2+2) = 496 test cases among the rest of operations based on their occurrence probabilities.

44 Allocating Test Cases Example /2 Example: Occurrence probabilities for normal operation mode. Infrequent operations below threshold Critical operation Table from Musa s Book

45 Allocating Test Cases Example /3 ~ 500 Number based on occurrence probabilities Infrequent operations below threshold Table from Musa s Book Critical operation

46 Structure of Lecture 4 Types of reviews Defect estimation Usage-based testing Certification testing

47 Question How to decide that a component (entity) has sufficient quality? In the following: Focus on the Quality Characteristic Reliability Typical application: Components-Off-The-Shelf (COTS) software ( 3 rd party software)

48 Reliability Certification Testing Process 5 Steps: 1. Define the reliability objective 2. Define the usage model and usage profile (operational profile) 3. Specify test cases 4. Execute certification test 5. Certify software component

49 Reliability Objective l obj Usually, the reliability objective l obj is defined as the desired maximal level of failure intensity (l F ) encountered during operation Failure intensity (l F ) is the inverse of Mean-Time- Between-Failure (MTBF) In the context of certification testing, failure intensity can be measured in terms of number of failures per test intensity (or test time or test effort) unit Example test intensity units: e.g. CPU hour, test person hour, number of test cases, etc.

50 Reliability Objective l obj Examples Typical values of reliability objectives are listed below; they are derived from the estimated impact (damage expressed in terms of $, and in terms of number of deaths) induced by a failure (Musa, 1998).

51 (observed) Reliability Demo Chart Musa (1977) Reliability goals are often stated in terms of Failure Intensity Objectives (FIO) Usually: Failure Intensity represents the number of Failures observed in a defined time period. Using a Reliability Demonstration Chart is an efficient way of checking whether the FIO (l obj ) is met or not. It is based on collecting failure data. Vertical axis: failure number (n) Horizontal axis: expected number of failures (or: normalized failure data (T n ), i.e., failure time l obj ) Expected number of failures Observed number of failures = Expected number of failures

52 How to Define Reject, Continue, Accept Regions? /1 The reject, continue, accept regions for a defined reliability objective (FIO) are based on sequential sampling theory. Procedure: 1. Select the discrimination ratio g with which the certification test will be performed; 2. Select the supplier (or developer) risk a, i.e. the probability of falsely deciding that the reliability objective is not met when it is; 3. Select the consumer (or customer) risk b, i.e. the probability of falsely deciding that the reliability objective is met when it is not.

53 How to Define Reject, Continue, Accept Regions? /2 n T n B lng n 1 g 1 g Boundary between reject and continue regions T n A lng n 1 g 1 g Boundary between accept and continue regions (g is the discrimination ratio) A b ln 1 a B 1 b ln a T n

54 Reliability Demo Chart Effects of a, b and g When risk levels (a and b) decrease, or When discrimination ratio (g) decreases, the system will require more testing before reaching the Accept or Reject regions i.e., the Continue region gets wider.

55 RDC: Example /1 Consumer risk b = 0.05 Supplier risk a = 0.05 Discrimination ratio g = 2

56 RDC: Example /2 Consumer risk b = 0.01 Supplier risk a = 0.01 Discrimination ratio g = 2

57 RDC: Example /3 Consumer risk b = Supplier risk a = Discrimination ratio g = 2

58 RDC: Example /4 Consumer risk b = 0.1 Supplier risk a = 0.1 Discrimination ratio g = 1.2

59 Example 1 Failure number Measure (million transactions) Normalized Measure (= expected Failure number) l obj = 4 failures / million transactions a = 0.1 b = 0.1 g = 2

60 Example 2 Failure number Measure (CPU hour) Normalized Measure (= expected Failure number) l obj = 0.1 failures / CPU hour a = 0.05 b = 0.05 g = 2

61 Example 3 We have developed a program for a Web server with a target failure intensity of 1 failure/1,000,000 transactions. The program runs for 50 hours, handling 10,000 transactions per hour on average, with no failures occurring. How confident are we that the program has met its objective? Can we release the software now? l obj = 1 failure / (10 6 transactions) a = 0.1 b = 0.1 g = 2

62 Example 3 Failure number Measure (transactions) Normalized Measure (= expected Failure number) 1? 500, ? 1,000, ? 3,000,000 3 l obj = 1 failure / (10 6 transactions) a = 0.1 b = 0.1 g = 2

63 Recommended Textbook Exercises Chapter 10 1, 5, 6, 7, 9, 11 Chapter 12 2, 3, 7

64 Next Week Lecture 5: Industry Presentation by Madis Jullinen: "Gaming as a gateway to better testing." Lab 4: Document Inspection and Defect Prediction In addition to do: Continue working on project Read textbook chapters 10 and 12 (available via OIS)