Faults Found by Random Testing. WanChi Chio Justin Molinyawe

Transcription

1 Faults Found by Random Testing WanChi Chio Justin Molinyawe

2 Introduction Why apply random testing to object oriented software? Cost-efficient in terms of implementation effort and execution time.

3 Introduction (Cont.) Characteristics of random testing Predictability in terms of number and nature of faults. Average time it takes to find a failure. Distribution of detected faults.

4 Introduction (Cont.) Experiments Predictability of Random Testing Creating and using tests with AutoTest for 27 classes from Eiffel library. Each class was tested for 90 minutes. For each class, testing sessions were ran 30 times with different seeds.

5 Unit Tests for Object- Oriented Programs Purpose: to verify the run of a particular routine: test routine "test_x" for routine "x" Hard-coded stubs must be used to replace non-trivial routines which are not under test for a strict interpretation, but it does not usually happens in practice

6 Unit Tests for Object-Oriented Programs (Cont.) Input are created in two ways by: loading previously saved objects into memory => to provide complete control over the input creation calling a series of constructors to create and modify data through normal means => to yield more relevant data

7 Unit Tests for Object-Oriented Programs (Cont.) Oracle/expect output can be provided in different levels of abstractions for validation through following specifications: The exact expected output - good for complex scenarios Conditions that the output should fulfill for any input "No exception" as the expected output - easier to be re-used for other tests

8 Random Tests Cases for Eiffel Library Experiment Testing is completely automated with test cases created under the following criterias: Unit - routines of Eiffel classes Test synthesizer makes test cases where the goal is to test each particular routine (Test Case Synthesis Algorithm) Input - Objects are created through regular constructor and routine calls. To enforce diversity, the initial objects are modified for new test cases

9 Random Tests Cases for Eiffel Library Experiment (Cont.) Oracle Contract-based by containing specifications in the form of routine pre and postconditions and class invariants

10 Faults and failures of Random Testing Usually, failures are an observed difference between the actual and intended behavior. Results relied on an approximation that maps failures to faults based on: Two failures are a consequence of the same fault if and only if they manifest themselves through the same type of exception.

11 Test Case Synthesis Algorithm Class C { R m(c 1 P 1,..., C n P n ); } A test case for routine m above can be executed as follows: c.m(a 1,..., a n ) => r where c = object of Class C r = output in Type R m = routine defined in Class C a j = argument object of Type C j for Parameter P j

12 Test Case Synthesis Algorithm (Cont.)

13 Using AutoTest Contract based testing tool. Allows easy plug-in of testing strategies. Composed of two processes Test Generator / Driver Interpreter AutoTest runs for a duration of the set time.

14 Number of Faults Will different runs of random testing with different seeds for the pseudo-random number generator reveal a similar number of faults? Will the faults be the same?

15 Number of Faults in Eiffel Library Experimental Setup 27 classes Each class was tested for 30 sessions of 90 mins Each session had a different seed to initialize the pseudo-random number generator. Total testing time was 30 * 27 * 90 = 50.6 days

16 Number of Faults in Eiffel Library Experimental Results Is random testing predictable? Two kinds of predictability: Number of faults Identity of faults and find if it is likely to detect the same faults regardless of the experiments was selected Normalized results give us aggregated results.

17 Number of Faults

18

19

20

21

22

23 Conclusion Results On average, 30% of the faults found during experiments are found after 10 mins. After 90 mins, 8% points, on average of the overall number of randomly detected faults are found. Relative to the number of detected faults, random+ is highly predictable. Different runs of the testing process reveal different faults.

24 Questions?