Offline Model-based Testing and Runtime Monitoring

Transcription

1 Offline Model-based Testing and Runtime Monitoring of the Sensor Voting Module Paolo Arcaini Angelo Gargantini Elvinia Riccobene Università of Bergamo- Italy Università di Milano - Italy Tolouse, ABZ 2014

2 Motivation For safety critical components, formal verification and validation of models must be combined with the validation of the implementation. confidence that the system has been implemented as specified, i.e., its implementation conforms to its requirements. regardless the correctness of the model (guaranteed by formal verification, simulation and so on), the implemented system must be validated itself. Even when formal refinement and/or program formal verification are applied Who would want to fly in an airplane with software proved correct, but not tested? Ed Brinksma in his 2009 keynote at the Dutch Testing Day and Testcom/FATES

3 General scenario Conforms to? requirements Formal specification Implementation Model-based testing Runtime Monitoring Models used for test data generation and verdicts computation (oracle) In off-line, test suites are derived from the model and saved. During runtime, checking whether a run of a system satisfies or violates a given correctness property

4 Proposed validation process 1.modelling ASM model 2 Model validation Req. 3. Implementing 4. Link Implementation and specification Java 5. Offline testing 6. Runtime monitoring Applied to the sensor voting module

5 1. ASM model for SVM Using AsmetaL function channel represents the signals coming from the three channels dynamic monitored channel: Channel > Boolean function validch keeps track if each channel is still valid dynamic controlled validch: Channel > Boolean the output is sensor and its validity is simply defined as a derived function valid dynamic controlled sensor: Boolean derived valid: Boolean RULES to set the value and the validity

6 2. Model validation 1. Simulation User guided Scenario guided all the sensors stay valid, the value is what expected sensor1 becomes invalid and after some steps also sensor2 fails We found at least one error in our initial specification 2. Formal Property Verification By model checking Once the sensor becomes invalid, then it will always remain invalid: CTLSPEC AG( not(valid) implies AG (not(valid))) There exists a path in which the sensor eventually becomes invalid: CTLSPEC EF (not(valid)) There exists a path in which the sensor always remains valid: CTLSPEC EG (valid)

7 3. Write the Java implementation The ASM and the Java implementation have been developed independently once we have agreed upon the interface A method to pass the current values of the channels A way to get the sensor value and validity More freedom (thanks to Java) Use of generics System.out / log4j The Java class can contain further methods not interested by the spec

8 4. Linking a Java class with its ASM model We reuse the approach CoMA Conformance Monitoring by ASMs Paolo Arcaini and Angelo Gargantini and Elvinia Riccobene CoMA: Conformance Monitoring of Java programs by Abstract State Machines in 2nd International Conference on RUNTIME VERIFICATION RV 2011 Based on the use of Java annotation (@) The implementation itself contains the definition of the links to the specification The link should be maintained during software evolution

9 4. Linking a Java class with its ASM model Given a Java class C and its ASM model Asm C 1. Link the class its specification Asm Specification Asm class Sensor {. } 2. Link the state Java state Abstract State 3. Link the execution Java methods link Java Class C

10 4b. Java observed state to ASM signature Java observed state OS(C) consists of all public fields, and pure non void public methods of the class C the user wants to observe link: OS C functions(asm C ) For the voting module: class Sensor sensor") public boolean getvalue(){ Asm Specification Asm C Signature Java Class public boolean isvalid(){ return sensorvalid;} OBSERVED STATE

11 4c Java methods to ASM execution the execution of the Java code must be linked with an execution(i.e., a run) of the ASM. Identify Java changing methods CM(C) is the set of methods of C whose execution is responsible for changing values of OS(C) and that the user wants to observe class public Sensor() { public void computesensorvalue() { Asm Specification Asm C Rules Java Class C Changing Methods

12 5. Offline testing ASM model Java Test generation TESTS Test concretization Test generation Using model checking (Spin) (ATGT) and several coverage criteria to generate abstract tests Test concretization The test are adapted to concrete test execution

13 5a. Test generation ATGT exploits the counter example generation of the model checker SPIN ATGT has derived 38 test predicates 20 for the BRC and 18 for the URC Each test predicate represents a test goal or test requirement For example valid = false : the sensor becomes invalid ATGT produces 38 test sequences 11 test sequences if tests reduction is applied They are enough to cover all the test goals Example of test sequence state 0 controlled valid = true monitored channel(one) = false channel(two) = false channel(three) = true state 1 controlled sensor = false valid = true

14 5b. Test concretization ATS - ABSTRACT TEST SEQUENCE Test are translated to JUnit code Deriving a concrete Java test A sequence of method calls with suitable checks (i.e., asserts) novel technique The test concretization leverages the linking between the Java class and the ASM and the definitions of state conformance and step conformance of point 3

15 Test translation to create a new Object For each state of the ATS, call the method annotated (+ parameters) Build the oracle by exploiting the state 0 controlled valid = true monitored channel(one) = false channel(two) = false channel(three) = true state 1 controlled sensor = false valid = public void test() { // state 0 Sensor sut = new Sensor (); assertequals(true, sut.isvalid()); sut.computesensorvalue(false, false, true); // state 1 assertequals(false, sut.getvalue()); assertequals(true, sut.isvalid()); }

16 6. Runtime Monitoring (CoMa) ASM model 1 2 O C - Java SIMULATOR OBSERVER 3 4 ANALYZER OK/FAIL 1. During the execution of the Java object of class C 2. an observer evaluates when the observed state is changed Using AspectJ 3. makes the abstract ASM to perform a machine step Using AsmetaL simulator 4. An analyzer evaluates the step conformance between the Java execution and the ASM simulation

17 Runtime conformance Runtime conformance O C is runtime conformant to its Asm C if: 1. no specification invariant of Asm C is ever violated. 2. the initial state of the computation of O C conforms to the initial state of the computation of Asm C 3. every observed change step conforms with the step of Asm C computation (states) Asm C O C Non changing methods steps are not

18 Experimental result Sensor Voting MBT: 38 Junit tests CoMA: we have simulated the environment by instantiating 10 times a new sensor and computing 10 times the sensor value passing three random values as inputs for the three channels. Evaluation Code coverage by EclEmma: 100% in both cases Mutation score by PIT 57 killed mutants over 74 in both cases Both techniques are equivalent regarding detecting faults inserted by the standard PIT mutation operators. CoMA has a great runtime overhead (ASM simulator must run in parallel)

19 When runtime monitoring is better? we have simulated a delayed short circuit fault that causes isvalid to return true after 5 times it is called. int nvcount = 0; boolean isvalid() { return valid nvcount++ > 5;} The tests produced from the specification do not detect this fault the rule coverage of the specification does not imply the coverage of this faulty behavior in the implementation. However, monitoring the code with CoMA exposes the failure by any run in which valid becomes false and isvalid is called at least 5 times. In general, unspecified anomalous behaviors of the implementation are better detected by runtime monitoring than by MBT

20 Conclusions A complete ASM-based methodology for the validation of code Guidelines/methodology for: Linking the code and its specification Deriving test cases from models Performing runtime monitoring Lesson learned Case study very useful for students/practitioners Importance of a linking (for both MBT and runtime, but also future use) This can limit the use of Java (skeleton generation could be useful) Future work: compare w.r.t other approaches like Junit tests written by hand/ JML - Design by Contract/ test generation -