Regression testing for the effective upgrade of an automatic track protection system

Transcription

1 Regression testing for the effective upgrade of an automatic track protection system Dr. Petr Štěpán Ph.D. Czech Technical University in Prague AŽD Praha

2 Zero on-site testing is very important for effective deployment of new products is intensively researched by many companies and project (Shift2Rail) till now at least reduce on-side testing to minimum Zero on-site testing

3 How to reduce on-site testing? Use intensive factory tests Use simulators to perform tests of final systems Use formal methods to prove system correctness Use model of the system Zero on-site testing

4 AŽD approach Use simulator to perform more tests than it can be done on-site The main problem is how to prove system correctness: Formal model is very complex and hard to develop Regression testing is using old version of product as a formal model This approach was tested on line signalling system Electronic automatic block system ABE-1 AŽD approach

5 Structure of the system: CENJ central unit core of the system EDOS control 3 block section signals EDOK control Continuous train Protection system coding EDON control 8 inputs/outputs DC voltage interface to station control systems EDOR control data interface to station control system KOMJ communication unit transmit data between stations Electronic automatic block system ABE-1

6 We are testing: Logical function of line block system 5 modules in C language, 3278 LOC, 49 functions, 7072 branches Correct response for peripheral errors (broken light bulb/led, communication errors, HW errors of peripheries) We are not testing (till now) CENJ communication subsystem Failure of HW (CPU, memory) Test plan

7 Testing in several steps: Compare the old real system with the simulated old system Compare the old simulated version with the new simulated version (if wrong behaviour is detected the new version is updated) Compare the new real version with the new simulated system Test plan

8 CENJ CENJ Simulated environment Test scenario Simulated CENJ Comparing real and simulated system Precise timing, one cycle takes 200ms Simulated CENJ is using the same input as real CENJ Compare CENJ outputs and internal states from diagnostics Real system testing

9 Simulated CENJ new version Simulated environment Test scenario Simulated CENJ old version Comparing simulated systems Can be faster than realtime We can compare CENJ s all internal states from simulation We can monitor the source code coverage Regression tests

10 Advantage using simulated system We can speed up simulated test in our case we can perform simulated tests 20 times faster We can analyse the code coverage by test cases We can design test cases to cover all lines of code Simulated tests

11 What does it means all lines of code? Part of code depends on system configuration We are performing test sets for different configuration Defence programming techniques are using code for detection of internal system failures Defence code lines in program are marked with a comment that they should not be performed The test checks whether the line has been executed These lines are added to coverage report if the lines were not executed Code analysis

12 GUI for simulated environment is written in Java Simulation is tested using Linux OS/MS Windows GNU gcc compiler is used for simulated CENJ (real system is using Keil compiler) Gcov for source code coverage Our application for analysis of lines, that should not be executed Lcov graphical front-end Used tools

13 Our results of source code covered: 100% function coverage 96 % of all statement coverage 100% statement coverage of the most important functions 86% branch coverage 98% branch coverage of state machines and we prove that the rest cannot be accessed 92% branch coverage of important functions We cannot prove, that the new version is completely error-free, but we significantly reduce the probability, that the system contains error Code analysis results

14 Our testing leads to zero on-site testing for upgrade of new version of CENJ This testing is far deeper than the former onsite testing Out test cases contains standard on-site test that were previously used This testing reduce money for system upgrade Conclusions