How to get to 100% code coverage

Transcription

1 How to get to 100% code coverage Intelligent Testing 2015 Bristol, 18 th June 2015 Dr Guillem Bernat

2 Introduction Code coverage is important Sometimes 100% is difficult What do we do if we can't get 100%

3 Code coverage defined

4 What does it mean to get 100% code coverage?

5 100% code coverage does not mean: your code is correct Testing shows the presence of bugs, not their absence Edsger Dijkstra (1969)

6 100% code coverage does not mean: your software requirements are correct

7 100% code coverage does not mean: the compiler translated your code correctly const volatile int x; volatile int y; void foo(void){ for(y=0; y>10; y++) { int z = x; } } At the -0s optimization level, GCC for IA32 compiles the C code on the left to the assembly on the right. The compiler output is wrong: the access to volatilequalified variable x should not be hoisted out of the loop. foo: mov1 $0, y mov1 x, %eax jmp.l2.l3: mov1 y, %eax incl %eax mov1 %eax, y.l2: mov1 y, %eax cmp1 $10, %eax jg.l3 ret

8 100% code coverage does not mean: you have covered 100% of your object code Source code Compiled to Object code

9 Does it matter which code coverage criteria I'm using? Coverage type DO-178B/C ISO (Software architecture) ISO (unit test) Function Used with MC/DC ASIL C, D Highly Recommended ASIL A, B Recommended - Call Coverage Not required ASIL C, D Highly Recommended ASIL A, B Recommended - Statement Level A, B, C Required - ASIL A, B Highly Recommended ASIL C, D Recommended Decision Level A, B Required - - Branch Not required - MC/DC Level A Required - ASIL B, C, D Highly Recommended ASIL A Recommended ASIL D Highly Recommended ASIL A, B, C Recommended

10 What are the barriers to 100% code coverage? Missing requirements. Missing or incorrect tests. Dead code. Deactivated code. Defensive programming. Impossible combinations of events. Compiler-introduced errors.

11 Barrier #1: Missing Requirements What to do about it Add new requirements Develop tests Run the tests

12 Barrier #2: Missing/incorrect tests What to do about it it Implement additional tests Ensure that they trace to the correct requirement(s) Run the new tests

13 Barrier #3: Dead code What to do about it Remove the dead code if possible or foo(a,c){ int b; What to do about it mode = MOVING; if(mode == STATIONARY){ b = 1 + a; } DO-178C recommends "The [dead] code should be removed and c an = a; analysis performed to assess the effect and the need for return reverification." c; [ c] } Justify why the code cannot be run For ISO 26262, the general guidance for "insufficient" coverage applies, namely to provide a rationale for why the dead code is acceptable [6:9.4.5].

14 Barrier #4: Deactivated code settings = read_input(config_pins); if ((settings & CONFIG_MASK_A) == CONFIG_A) { /* code to execute in configuration A */... } else if ((settings & CONFIG_MASK_B) == CONFIG_B) { /* code to execute in configuration B */... } What to do about it Justify why code won t be executed Never executed during normal operation Will not affect the execution of the system

15 Barrier #5: Defensive Programming switch(a % 4){ case 0: } case 1: case 2: case 3: default:... break;... break;... break;... break;... break; What to do about it Justify why defensive programming cannot be executed Provide evidence that if triggered, the defensive programming works correctly

16 Barrier #6: Impossible Combinations of Events What to do about it -- enabled might already be set at this point if speed < 1700 then enabled := true; end if; Simplify the condition? or Justify why 100% could not be achieved if speed < 1500 and enabled then... end if;

17 Barrier #7: Compiler Introduced Errors int a,x; if a < 10{ /*Compiler generates a <=10 */ result = x / 5; } else { result = x / 10; } What to do about it Review existing code base Add to coding guidelines / code review guidelines a x Test case Test case

18 Justifying code: in-line justifications void fatal_error (void) { #pragma RVS justification( "COV_FUNCTIONS", "The fatal_error function cannot be executed during normal execution") /* some code */ }... void check_status (int status) { if ( (check_enabled) && (FATAL_ERROR == status) ) { #pragma RVS justification( "COV_DECISIONS", "Variable 'status' cannot take value FATAL_ERROR during normal execution") fatal_error(); } }...

19 Justifying code: justifications in a separate file #pragma RVS [selftest.c:34] justification("cov_mcdc", "2: (RAM_ERROR == status)", "RAM_ERROR cannot be triggered during normal test"); #pragma RVS [selftest.c:439] justification("cov_functions", "Cannot trigger emergency_shutdown() during normal test");

20 Justifications editor

21

22 Dealing with less than 100% code coverage Review establish missing coverage Identify cause Identify action to take

23 Dealing with less than 100% code coverage Review establish missing coverage Identify cause Identify action to take Missing requirements. Missing or incorrect tests. Dead code. Deactivated code. Defensive programming. Impossible combinations of events. Compiler-introduced errors.

24 Dealing with less than 100% code coverage Review establish missing coverage Identify cause Identify action to take Remove code Add (or fix) requirements Justify why code won't be executed Justify why it isn't possible to test

25 Structural coverage analysis to DO-178B/C Level A and ISO reduce testing timescales by running fewer on-target tests reduce risk through greater tool flexibility reduced effort for certification activities

26 Logos and images can be positioned within this area. Change layout to blank (without position box) before use

27 Q A