Testing, code coverage and static analysis. COSC345 Software Engineering

Transcription

1 Testing, code coverage and static analysis COSC345 Software Engineering

2 Outline Various testing processes ad hoc / formal / automatic Unit tests and test driven development Code coverage metrics Integration testing Limits of testing Testing and continuous integration Static analysis symbolic execution as an example!2

3 Ad hoc / formal / automated testing All developers apply ad hoc testing of some sort By definition it s necessary by running and using your code More formal testing approaches will other benefits: Make it easier to protect against regressions in code (Regressions: losses in functionality from bugs in new functionality) Give another means of documenting functionality of code Automated tools can increasingly do testing for you: Increasingly complex simulations accelerate bug discovery!3

4 Unit tests Divide code up into units and built sets of tests per unit: Want tests to be independent of each other, where possible Want coverage of tests of unit s functionality to be complete! Tested code should behave as if in production, even if it is interacting with external resources, e.g., databases Thus, tests can usefully be run in some sort of framework that helps linking in stubs or proxies for external interactions although this means that another model is being included that captures the behaviour of those external resources!4

5 Object oriented languages and unit tests Objects in OO PLs can be good targets for unit tests For other such targets, think back to lectures on components Language facilities can help support testing needs e.g., polymorphism allows hiding test / production differences Objects provide a boundary for state and behaviour e.g., can provide a proxy database handle to a class Well-designed objects will have workable code length OO refactoring should lead to methods with fewer LoC!5

6 Costs and benefits of unit testing Cost: testing can involve many LoC and takes time ( but how long were you planning to be debugging code?) On the other hand, there are many benefits Can remove bugs prior to the system integration stage Baseline against which to check refactoring safety Baseline against which to detect regressions Augments API documentation by indicating key concerns!6

7 Test-driven development Write the tests before you extend your implementation You should explicitly test that these new tests fail! A key goal is to to turn requirements into tests Provides a contract against which to develop code Contract is also minimal: no need for more than passing tests Most TDD approaches also enshrine refactoring stages Of course tests are run during the refactoring process too!7

8 Code coverage of tests (AKA test coverage) Examines what proportion of code is actually tested Miller and Maloney published a paper about this in 1963 Do you really want to run code that s had no testing? Many coverage metrics with different ease / tooling Function has every method / function / routine been called? Branch have all paths from conditional statements been run? Statement has each statement been run at least once? Condition have all expressions evaluated to true and false?!8

9 Target code coverage values Increasing code coverage gets progressively harder how much harder depends on the type of code being tested Safety critical systems may require 100% coverage although there will be caveats to what coverage means There are many good heuristics to use to prioritise testing: Loops have been skipped; run once; and run many times Data flow of variables has been explored, not just code focus All possible entry & exit points of a function have been tested!9

10 Integration testing (after unit testing) Integration testing involves tackling the complexity of interacting parts of code so more complex to write Unit tests are intentionally insulated from the environment Integration tests fail due to changes in environment The business logic should already be tested within unit tests Integration tests are likely to require more time to run May need to set up + tear down multiple resources, e.g., DBs!10

11 Limits of testing Cannot test all possible situations (see code coverage) To do so would defy a generalisation of Turing s halting problem i.e., it s not possible to build a single program f that decides whether any given program g and input to it x will halt or not Different degrees of testability have been classified: Class I: code can be completely tested with a finite test suite Class II: a finite test suite can t tell [in]correct systems apart Class III: there is a countable complete test suite Class IV: there is a complete test set Class V: everything else (e.g., non-deterministic functions)!11

12 Testing and Continuous Integration Most forms of automated testing are good candidates for use within continuous integration frameworks Good to know tests are passing for each code commit e.g., build badges on project websites For production code release, it almost always makes sense to require passing all tests in the test suite this is enforced easily CI can run both unit tests and integration tests Many CI systems allow such tests to be collected into phases e.g., less point deploying integration test resources if unit tests fail!12

13 Test that you can rebuild from a clean start Continuous integration systems will typically require this but some can support caching various resources for speed e.g., Docker-based CI tools may cache container images Important that you are confident your project can be rebuilt using your build process from a clean start For local git repositories I will often `git clone` a new working copy nearby my actual working copy, to test!13

14 Static analysis a type of automatic testing Static analysis examines code without running it We saw SpotBugs in tutorials: it employs static analysis Protection was provided against null pointer errors (to a point) also protection against resource allocation errors Vast range of complexity levels in static analysis tools: Simple: apply linting approaches to highlight worrying code Complex: formal methods that simplify and model code effect e.g., data-flow analysis, abstract interpretation, symbolic execution!14

15 Symbolic execution is a static analysis tool Symbolic execution tracks constraints on variables Does not require any instrumentation of the source code However it can quickly be overcome by code path structure Consider code: if(x<20){ A } else { B } Forks analysis down both branches: on A x<20, on B x>=20 e.g., if(x<4) inside A would be run since symbolic x is <20 Likewise a loop over a string will gather array constraints e.g., a 2-char C-string s will include s[0]!=0; s[1]!=0; s[2]=0!15

16 Symbolic exec: KLEE Found 9 errors in GNU Coreutils; 3 were 15+ years old A MINIX tr command bug is in these listings int main(int argc, char* argv[ ]) { int index = 1; if (argc > 1 && argv[index][0] == '-') { /*...*/ } /*...*/ expand(argv[index++],...); /*...*/ } void expand(char *arg, unsigned char *buffer) { int i, ac; while (*arg) { if (*arg == '\\') { arg++; i = ac = 0; if (*arg >= '0' && *arg <= '7') { do { ac = (ac << 3) + *arg++ '0'; i++; } while (i < 4 && *arg >= '0' && *arg <= '7'); *buffer++ = ac; } else if (*arg!= '\0') *buffer++ = *arg++; } else if (*arg == '[') { arg++; i = *arg++; if (*arg++!= '-') { *buffer++ = '['; arg = 2; continue; } ac = *arg++; while (i <= ac) *buffer++ = i++; arg++; /* Skip ']' */ } else *buffer++ = *arg++; } }