Test Oracles and Randomness

Transcription

1 Test Oracles and Randomness UNIVERSITÄ T ULM DOCENDO CURANDO SCIENDO Ralph Guderlei and Johannes Mayer rjg@mathematik.uni-ulm.de, jmayer@mathematik.uni-ulm.de University of Ulm, Department of Stochastics and Department of Applied Information Processing Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 p. 1

2 Introduction Standard methods of software testing do not provide information about software reliability. random testing The main problem in random testing is to verify the actual results of the Implementation Under Test (IUT). Oracles Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 p. 2

3 Oracles An oracle provides methods to generate expected results for test cases to compare the expected results to the actual results of the IUT. Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 p. 3

4 Oracles An oracle provides methods to generate expected results for test cases to compare the expected results to the actual results of the IUT. It consists of two parts: the result generator to obtain expected results the comparator to verify the actual results of the IUT Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 p. 3

5 Standard Types of Oracles Oracles do not apply generally, only in special cases. Standard types are Perfect Oracle Gold Standard Oracle Parametric Oracle or Heuristic Oracle Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 p. 4

6 Perfect Oracle equivalent to the IUT and completely trusted accepts every input specified for the IUT produces always the correct result a defect free version of the IUT Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 p. 5

7 Gold Standard Oracle Test Input Test Case Input Golden Implementation golden result Comparator Pass / no Pass IUT actual result Use one or more versions of an existing application system to generate expected results (e.g. a legacy system). Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 p. 6

8 Parametric Oracle Test Input IUT actual results Output Converter actual parameters Trusted Algorithm reference parameters Comparator Pass / no Pass Use an algorithm to compute parameters from the actual results and compare the actual parameters to expected parameter values. Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 p. 7

9 Statistical Oracle special case of a parametric oracle Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 p. 8

10 Statistical Oracle special case of a parametric oracle parameters are computed with statistical tools Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 p. 8

11 Statistical Oracle special case of a parametric oracle parameters are computed with statistical tools comparison is done in a statistical way Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 p. 8

12 Statistical Oracle special case of a parametric oracle parameters are computed with statistical tools comparison is done in a statistical way generation of random input data allows a large number of test cases Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 p. 8

13 Micropattern Random Test Input Generator Test Case Input IUT Actual Results Statistical Analyzer Characteristics Distributional Parameters Comparator Pass / No Pass distributional properties of the generator and the IUT are used for the comparison Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 p. 9

14 Requirements statistical characteristics of the IUT have to be known large number of input data for stable results (> 30) Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 p. 10

15 Possible Uses (Scientific) applications dealing with randomness e.g. simulators, data analysis (e.g. in banking, image analysis) Applications with complicated input data, where reference values are difficult to obtain. As in the Example: images are difficult to analyze randomly generating and comparing the mean values is simple Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 p. 11

16 Necessary Statistics X i iid random variables with mean µ and variance σ 2 Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 p. 12

17 Necessary Statistics X i iid random variables with mean µ and variance σ 2 sample mean of n random variables: n X n = 1 n X i i=1 Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 p. 12

18 Necessary Statistics X i iid random variables with mean µ and variance σ 2 sample mean of n random variables: n X n = 1 n X i i=1 sample variance of n random variables: n Sn 2 = 1 n 1 (X i X n ) 2 i=1 Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 p. 12

19 Distributional Properties using the central limit theorem, one can assume the X i to be (assymptotically) normally distributed. n Xn µ σ d N(0, 1) as n Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 p. 13

20 Simple Approach expected value is known: µ 0 Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 p. 14

21 Simple Approach expected value is known: µ 0 compute empirical mean of the n actual results x i : x n = 1 n n i=1 x i Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 p. 14

22 Simple Approach expected value is known: µ 0 compute empirical mean of the n actual results x i : x n = 1 n n i=1 x i pass if x n µ 0 µ 0 < ε, e.g.ε = 0.1 Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 p. 14

23 First Attempt Natural choice: t-test if the mean of the actual results µ is equal to the expected result µ 0. Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 p. 15

24 First Attempt Natural choice: t-test if the mean of the actual results µ is equal to the expected result µ 0. Type I error: IUT does not pass though it is correct (false alarm) Type II error: IUT does pass though it is not correct (false pass) But only the Type I error will be controlled. Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 p. 15

25 Advanced Approach Alternative choice: use the intersection-union method to invert the test hypothesis. So the controllable probability for the Type I error becomes the probability for a false pass. Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 p. 16

26 Advanced Approach Alternative choice: use the intersection-union method to invert the test hypothesis. So the controllable probability for the Type I error becomes the probability for a false pass. With δ > 0, one can define an interval around µ 0 such that X n / [µ 0 δ, µ 0 + δ] for a given probability α. Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 p. 16

27 The Test Statistic For a given probability α (0, 1 2 ), the IUT passes if and n x n (µ 0 δ) s n n x n (µ 0 +δ) s n t n 1, α 2 t n 1, α 2, Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 p. 17

28 The Test Statistic For a given probability α (0, 1 2 ), the IUT passes if and n x n (µ 0 δ) s n n x n (µ 0 +δ) s n t n 1, α 2 t n 1, α 2, where t n, α denotes the (1 α/2) - quantile of the 2 Student t-distribution with n 1 degrees of freedom. Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 p. 17

29 An Example from Image Analysis compute morphological properties such as area or boundary length fit stochastic models to the given data Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 p. 18

30 Random Input - The Boolean Model computationally simple model flexible good fit to real data the expected mean area and mean boundary length are known in explicit form Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 p. 19

31 Micropattern Revisited Generator for Boolean Model Test Case Input IUT Actual Results, e.g. Area Statistical Analyzer Characterstics, e.g. Mean Area Theoretical Results from the Model Comparator Pass / No Pass Random Input Generator = Generator for the Boolean Model The IUT computes e.g. the area or the boundary length Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 p. 20

32 Usage of the Simple Approach Performing the test for each computed characteristic (e.g. the area) for different Boolean Models Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 p. 21

33 Usage of the Simple Approach Performing the test for each computed characteristic (e.g. the area) for different Boolean Models The simple approach was used as advanced smoke test to detect severe bugs in the program flow plausibility check for the computed values Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 p. 21

34 Usage of the Advanced Approach verification of the results of the simple approach choosing a small α, one can say that the IUT produces the correct results (only) with respect to the tested characteristics with probability α Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 p. 22

35 Numerical Example sample of n = 500 images, error probability α = , here x i ˆ= boundary length Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 p. 23

36 Numerical Example sample of n = 500 images, error probability α = , here x i ˆ= boundary length x n = , µ 0 = , δ = = 1% µ 0 Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 p. 23

37 Numerical Example sample of n = 500 images, error probability α = , here x i ˆ= boundary length x n = , µ 0 = , δ = = 1% µ 0 n x n (µ 0 δ) s n = > 3.92 = t 499, and n x n (µ 0 + δ) s n = < 3.92 = t 499, Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 p. 23

38 Numerical Example sample of n = 500 images, error probability α = , here x i ˆ= boundary length x n = , µ 0 = , δ = = 1% µ 0 n x n (µ 0 δ) s n = > 3.92 = t 499, and n x n (µ 0 + δ) s n = < 3.92 = t 499, So the IUT passes the test. Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 p. 23

39 Conclusion The approach includes information about the reliability with respect to for the tested characteristics The approach makes it possible to handle randomness in tests It is possible to test for other characteristics than the mean The approach shown above does not apply in all cases Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 p. 24

40 Thank you for your attention. Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 p. 25