Requirements satisfied : Result Vector : Final : Matrix M. Test cases. Reqmts

Transcription

1 Introduction Control flow/data flow widely studied No definitive answer to effectiveness Not widely accepted Quantitative measure of adequacy criteria Effectiveness Whether cost of testing methods is justified Paper addresses these issues Overview Empirical evaluation Subject programs Large number of test sets Coverage of testing criteria Effectiveness Result Analysis Comparison/Relationships Previous studies Small programs Seeded faults This paper 8 subject programs Antenna configuration program 10, 000 lines of C code Real faults Experiment Design Consider Faulty program P Specification S Adequacy criterion C on P and S. Large number of test sets satisfy C Differences in ease of execution, effectiveness, etc. Need many test sets Statistical techniques Design Subject programs Faults discovered, isolated Correct base program In each version, one fault» Real faults» Low failure rates Test universe generation For each program, universe of 10, 000 test cases Generator developed by Pasquini et al 1 Randomly choose test cases, make test sets 1

2 Coverage matrix M Entries» M(i,j) = 1, test case i covers requirement j» = 0, i does not cover requirement j ATAC 2 software testing tool Results vector R Entry for each test case Whether test case detects at least 1 fault Simulate test set of size s Randomly select s rows of the matrix M OR them together» Requirements covered Combine with results vector R» Faults exposed Maintain 2 arrays, total and exposing Determine level c of test set Increment total[c] for that level If fault trapped Increment exposing[c] for that level Reqmts Test cases Test case 1: Test case 2: Test case 3: Matrix M Branch 1 Branch 2 du pair 1 du pair Requirements satisfied : Result Vector : Final : Repeat for very large number of test sets Determine e.g. total[10% dec] ++ total[20% dua] ++ exposing [10% dec] ++ Estimate effectiveness for a given level c n c = S i>=c total[i] x c = S i>=c exposing[i] p^c = x / n c c p^c is estimate of effectiveness 2

3 Confidence levels Statistical approximation methods = 1.96 [(p^c (1 - p^c ) / n c ) ]0.5 e c Benefits of Design Allows control on test set size s Previous work s varied Comparison with random testing Probability that true value of p^c ( p^c - e c, p^c + e c ) is less than 5% lies outside Table 1: Subject Programs Results? - Decision - All-use Vertical bars - 95% confidence intervals Results Graphs in the preceding 2 versions very similar Faults are quite similar Both fail on same test cases 3

4 4

5 Results Mostly, effectiveness higher with higher levels. In one version, effectiveness falls at highest In previous work 3, also observed Results outside confidence interval? Or other factor? Guaranteed/ dependable/cost - effective?.. Results In some, at high, decision better Is this conclusive? Decision/dua perform similarly. Sharp contrast to earlier work with smaller subject programs 3.. Results Mostly, sharp increase in effectiveness with increase in levels. Perhaps, benefits of these criteria only at high Random test behaviour can be approximated Effectiveness found to be lower than our adequacy criteria Threats to validity Subjects from same project Results dependant on specific universe If universe not representative, results are biased Fixed test set sizes Correlation between results and test set sizes Occasionally, problems with ATAC Random test selection Conclusions Tests cases satisfying high decision/dua more effective than randomly sampled test cases Decision/All-use comparison not conclusive Bibliography 1. A. Pasquini, A.Crespo and P.Matrella Sensitivity of reliability-growth models to operational profiles errors v. testing accuracy. IEEE Transactions on Reliability. 2. J. Horgan and S. London. Data flow and the C language. IN Proceedings Fourth Symposium on Software Testing, Analysis, and Verification. 3. P.G. Frankl and S.N. Weiss. An Experimental comparison of the effectiveness of branch testing and data flow testing. IEEE Transactions on Software Engineering. Small step towards answering question of effectiveness of testing criteria Scope for further testing and more work 5