Automatic Test Suite Amplification for DevOps

Transcription

1 Automatic Test Suite Amplification for DevOps Benoit Baudry KTH, Sweden 1

2 DevOps 2

3 DevOps: Automatic software engineering unit perf. UI fuzzing dep. inj. IDEs libraries container CI app Integration logging IDS VMs chaos eng. cluster fault recov. 3

4 Automatic Testing in DevOps unit perf. chaos eng. UI fuzzing CI Integration logging fault recov. 4

5 Test Amplification Amplification is about automatic transformation of existing test assets in order to improve their fault detection capacities and drive down the cost of software testing. 5

6 Today: unit test amplification unit 6

7 Unit testing P Input space Observation space 7

8 Unit testing P state Input space Observation space 8

9 Unit test amplification P state Input space Observation space 9

10 Original test case public void testiterationorder() { for (int size = 1; size < 1000; size++) { } List<Integer> other = new ArrayList<Integer>(size); for (int i = 0; i < size; i++) {other.add(i);} TreeList<Integer> l = new TreeList<Integer>(other); ListIterator<Integer> it = l.listiterator(); int i = 0; while (it.hasnext()) { Integer val = it.next(); } Benoit } Baudry, KTH, ICES DevOps assertequals(i++, val.intvalue()); 10

11 public void testiterationorder() { } Original test case for (int size = 1; size < 1000; size++) { } List<Integer> other = new ArrayList<Integer>(size); for (int i = 0; i < size; i++) {other.add(i);} TreeList<Integer> l = new TreeList<Integer>(other); ListIterator<Integer> it = l.listiterator(); int i = 0; while (it.hasnext()) { Integer val = it.next(); } assertequals(i++, val.intvalue()); input (data and functions) oracle (expected properties) 11

12 Outcome of amplification 12

13 DSpot P TS I-ampl A-ampl Qualify 13

14 Search-based input space amplification Mutate existing test cases transform literal values replace literal values by others found in the scope add new invocations to methods not well covered create new objects Search-based stack transformations iteratively select the ones that have a good fitness (e.g. cover new branches) 14

15 Observation space amplification Remove all assertions from original test cases keep invocations that can be inside the assertion Add new observation points run all test cases on original program record all values that can be observed (getters, is_*, etc.) Filter new observation points several runs of all test cases with observation points on the original program discard all points that vary (e.g., OS-dependent values, IO, etc.) 15

16 Example public void testsearch() { final ArrayStack<E> stack = makeobject(); stack.push(((e)("first Item"))); stack.push(((e)("second Item"))); assertequals("top item is \'Second Item\'", 1, stack.search("second Item")); assertequals("next Item is \'First Item\'", 2, stack.search("first Item")); assertequals("cannot find \'Missing Item\'", -1, stack.search("missing Item"));} public void testsearch_cf5() { final ArrayStack<E> stack = makeobject(); stack.push(((e)("first Item"))); stack.push(((e)("second Item"))); E o_5 = stack.peek(); assertfalse(((java.lang.string)o_5).isempty()); E o_6 = stack.pop(); new calls and asserts assertfalse(((java.lang.string)o_6).isempty()); Object o_7_0 = stack.search("second Item"); Object o_9_0 = stack.search("first Item"); extracted from asserts assertequals("cannot find \'Missing Item\'", -1, stack.search("missing Item")); assertequals(((java.lang.string)o_5).length(), 11); assertequals(((java.lang.string)o_6).length(), 11);} new asserts 16

17 Qualify the new tests Tests are good if they can detect bugs Principle: generate bugs and test the tests Descartes tool Strip out the body of methods 17

18 Descartes - example long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} 18

19 Descartes - example long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} long fact(int n){ return 1;} long fact(int n){ return 0;} 19

20 Mutation analysis Inputs : P, TS, Ops Output : score, coverage M <- generatemutants (P, OPs) forall (m in M) run (TS,m) if (one-test-fail) then killed <- m else alive <- m score = killed / size(m) 20

21 Test amplification Amplification can improve the mutation score of existing test suites I-ampl explores novel behavior A-ampl adds new observations Deemed valuable by developers 21

22 Feedback welcome!