Software Testing for Critical Systems

Transcription

1 Software Testing for Critical Systems Julien Fayolle and Sandrine-Dominique Gouraud 1 1 Génie logiciel, LRI, Université d Orsay. {fayolle, gouraud}@lri.fr Web pages: fayolle and gouraud University of York, CS seminar. Fayolle & Gouraud Software Testing for Critical Systems 1

2 Overview of Testing How does one test a software? Testing all entries is not possible. We don t prove the software right. Create a test set. Functional testing: specifications, black box Structural testing: structure and code, glass box Statistical testing: draw at random from the input domain Run the program on the test set. Compare the outputs to the expected outputs (oracle). Fayolle & Gouraud Software Testing for Critical Systems 2

3 Statistical Testing Software for critical systems (planes motors, nuclear plants safety, missiles,... ); Intensive statistical testing; Provide quantitative figures on the test set; Provide adequacy figures on the test methods; Drawback: coverage of rare cases. factorial(integer n) if(n!=0) fact:=1; for i from n to 2 fact:=fact*i; else fact:=0; Fayolle & Gouraud Software Testing for Critical Systems 3

4 Statistical Structural Testing First answer: [Thévenod-Fosse & Waeselynck 91, LAAS] Combining statistical and structural testing. Compute a distribution on the input domain and draw tests inputs accordingly. How representative is the test set? Fayolle & Gouraud Software Testing for Critical Systems 4

5 Test quality 1 6 a c g j 0 e 3 4 h 7 b f k 2 d 5 Test quality q n = 1 (1 q) n, where n is the number of tests and q the minimal probability to reach one element. Fayolle & Gouraud Software Testing for Critical Systems 5

6 AuGuSTe Drawback: the input distribution has to be computed by hand. AuGuSTe. Automated Generation of Statistical Tests, [Gouraud et alii 04] First programmed for C code. Coded in Objective Caml; uses GATEL, MuPAD, Eclipse, Prolog, C, and Java. Statistical structural testing. Drawing uniformly at random paths of a given length in the control graph. AuGuSTe web site: fayolle/auguste.html Fayolle & Gouraud Software Testing for Critical Systems 6

7 Structural coverage criteria Given a control flow graph All paths All i-paths Branch coverage: all edges Block coverage: all instructions Predicate coverages Test quality for statistical structural tests q n minimal probability to reach one element with a test set of size n. q n = 1 (1 q) n. Fayolle & Gouraud Software Testing for Critical Systems 7

8 Random generation of paths Goal: optimize the test quality i.e. maximize the smallest probability to reach an element. Combinatorial description of the control graph Uniform drawing of path (criteria consideration) 1 6 a c g j 0 3 e 4 7 b 2 d f h 5 k Atoms are edges G = ac(egj + fg 1 ) + bdg 1, and G 1 = hj + k. Fayolle & Gouraud Software Testing for Critical Systems 8

9 How does AuGuSTe work? Inputs Program Coverage criteria Number of tests (test quality) Maximal length of the paths (elementary path) 1. Analysis: building the combinatorial structure of the control graph 2. Drawing paths uniformly at random in the graph 3. Constraint resolution (GATEL) Output Test set. Fayolle & Gouraud Software Testing for Critical Systems 9

10 Testing methods comparison using mutants Creating mutants : seed faults in the original program. A large repertoire of mutation operators: changing in, > or <; changing 0 in 2; performance criteria: mutation score (kill count). stability of the fault detection ability average. Softwares for creating mutants Mothra [Offutt, DeMillo et alii 88]; SESAME [Crouzet, Thévenod-Fosse & Waeselynck 98]; µjava [Offutt, Ma & Kwon 03]. Fayolle & Gouraud Software Testing for Critical Systems 10

11 Experimental results Function from industrial software (77 lines) 605 mutants (for some mutations). Test quality runs of test sets of size 850. min avg max Uniform testing NA Structural statistical testing AuGuSTe Table: Experimental mutation scores over 5 runs Fayolle & Gouraud Software Testing for Critical Systems 11

12 GATEL GATEL Test Suites Generation from Lustre Descriptions [Marre et alii, CEA and LRI] Uses Lustre descriptions. Lustre code or Lustre spec. (test) Uses constraint solving techniques, non-deterministic heuristics. Structural or functional software testing. GATEL web site : Fayolle & Gouraud Software Testing for Critical Systems 12

13 Lustre code node Alarm(stream0, s up, s down: real) returns (alarm s: bool); var upstream, downstream: bool let upstream = stream0 > s up; downstream = stream0 < s down ; alarm s = Automaton(upstream, upstream, downstream); tel; node Automaton(init, upstream, downstream: bool) returns (state: bool); let state = init -> if upstream and not pre(state) then true else if downstream and pre(state) then false else pre(state); let; Fayolle & Gouraud Software Testing for Critical Systems 13

14 Lustre Declarative language for specification and/or programming. Reactive synchronous systems. Data-flow. The specification allows the automatic generation of code. Fayolle & Gouraud Software Testing for Critical Systems 14

15 Perspectives Extend the idea of uniform drawing to the synchronous data-flow languages. Application to Lustre Enrich GATEL with random generation of combinatorial structures. Provide precise bounds on the fault detection ability of AuGuSTe-like methods for imperative and declarative languages. Fayolle & Gouraud Software Testing for Critical Systems 15