1,000 Testers in a Shoe Box Advances in Automated Test Generation Andreas Zeller Saarland University, Saarbrücken, Germany

Transcription

1 1,000 Testers in a Shoe Box Advances in Automated Test Generation Andreas Zeller Saarland University, Saarbrücken, Germany Visual Computing Institute Center for Information Security, Privacy and Accountability

2 Testing

3 Software is manifold

4 Software is manifold

5 Software is manifold

6 Software is manifold

7 Software is manifold

8 Software is manifold

9 Testing Configurations

10 Testing Configurations

11 What to test? How do we cover as much behavior as possible? Configurations

12 Testing Criteria Path testing Boundary interior testing LCSAJ testing Branch testing subsumes Compound condition testing MC/DC testing Branch and condition testing Practical Criteria Theoretical Criteria Loop boundary testing Statement testing Basic condition testing

13 Dijkstra s Curse Testing can only find the presence of errors, not their absence Configurations

14 Formal Verification Configurations

15 Formal Verification Abstraction Configurations

16 Formal Verification Abstraction Configurations

17 Formal Verification Abstraction Configurations

18 This might crash static tree fold_negate_expr (location_t loc, tree t) { // 200 Lines... tem = TREE_OPERAND (t, 0); if t is null if ((INTEGRAL_TYPE_P (type) && (TREE_CODE (tem) == NEGATE_EXPR (TREE_CODE (tem) == INTEGER_CST && may_negate_without_overflow_p (tem))))!integral_type_p (type)) return fold_build2_loc (loc, TREE_CODE (t), type, negate_expr (tem), TREE_OPERAND (t, 1)); }

19

20

21 but can these ever be invoked from here? static analysis may detect errors here

22 ? This might crash This might crash This might crash This might crash This might crash This might crash This might crash This might crash This might crash This might crash

23 Zeller s Variation on Dijkstra Abstraction Verification can only find the absence of errors, but never their presence Configurations

24 The Best of two Worlds Abstraction Configurations

25 Generating Tests Abstraction Configurations

26

27 Generated Unit Test public class RandoopTest0 extends TestCase {...! public void test8() throws Throwable { if (debug) System.out.printf("%nRandoopTest0.test8"); AddressBook var0 = new AddressBook(); EventHandler var1 = var0.geteventhandler(); Category var2 = var0.getrootcategory(); Contact var3 = new Contact(); AddressBook var4 = new AddressBook(); EventHandler var5 = var4.geteventhandler(); Category var6 = var4.getrootcategory(); String var7 = var6.getname(); var0.addcategory(var3, var6); SelectionHandler var9 = new SelectionHandler(); AddressBook var10 = new AddressBook(); EventHandler var11 = var10.geteventhandler();

28 MainWindow var64 = new MainWindow(var0); AddressBook var65 = new AddressBook(); EventHandler var66 = var65.geteventhandler(); Category var67 = var65.getrootcategory(); Contact var68 = new Contact(); Category[] var69 = var68.getcategories(); var65.removecontact(var68); java.util.list var71 = var65.getcontacts(); AddressBook var72 = new AddressBook(); EventHandler var73 = var72.geteventhandler(); Category var74 = var72.getrootcategory(); EventHandler var75 = var72.geteventhandler(); SelectionHandler var76 = new SelectionHandler(); actions.createcontactaction var77 = new actions.createcontactaction(var72, var76); boolean var78 = var77.isenabled(); AddressBook var79 = new AddressBook(); EventHandler var80 = var79.geteventhandler(); Category var81 = var79.getrootcategory(); String var82 = var81.getname(); var77.categoryselected(var81); Category var85 = var65.createcategory(var81, "hi!"); String var86 = var85.tostring(); Category var88 = var0.createcategory(var85, "exceptions.namealreadyinuseexception"); }

29 Simplified Test Case public class RandoopTest0 extends TestCase { public void test8() throws Throwable { if (debug) System.out.printf("%nRandoopTest0.test8");! } } AddressBook a1 = new AddressBook(); AddressBook a2 = new AddressBook(); Category a1c = a1.createcategory(a1.getrootcategory(), "a1c"); Category a2c = a2.createcategory(a1c, "a2c");

30 how many addressbooks?

31 sub-category of in in public class RandoopTest0 extends TestCase { public void test8() throws Throwable { if (debug) System.out.printf("%nRandoopTest0.test8");! } } a1 a1c a2 a2c AddressBook a1 = new AddressBook(); AddressBook a2 = new AddressBook(); Category a1c = a1.createcategory(a1.getrootcategory(), "a1c"); Category a2c = a2.createcategory(a1c, "a2c");

32 112 failures 0 PROBLEMS

33 0 PROBLEMS

34 but can these ever be invoked from here? testing may detect errors here

35 test generation at the system level

36 test generation at the system level anything that happens here is real

37 Getting Coverage static tree fold_negate_expr (location_t loc, tree t) { // 200 Lines... tem = TREE_OPERAND (t, 0); if ((INTEGRAL_TYPE_P (type) && (TREE_CODE (tem) == NEGATE_EXPR (TREE_CODE (tem) == INTEGER_CST && may_negate_without_overflow_p (tem))))!integral_type_p (type)) return fold_build2_loc (loc, TREE_CODE (t), type, negate_expr (tem), TREE_OPERAND (t, 1)); } get me here

38 what to enter here? get me here

39 Infinite Monkey Theorem

40 Test Generation Static analysis and symbolic test generation work well at the unit level but miss the surrounding context System test generation generates real executions but misses features at the code level

41 Evolutionary Algorithms Create population Create mutations Rank Recombine Select

42 Create population fdsakfh+ew%3gfhdi%4f fwe8^ru786234jä Mutation fdsakfh+br%3gfhdi%4f fdsakfh+ew%4gfhdi%4f fwe8^ru&26234jä xb3#ru786234jä Recombine fdsakfh+ew%4gfhdi%4f xb3#ru786234jä

43 Create population fdsakfh+ew%3gfhdi%4f fwe8^ru786234jä Mutation fdsakfh+br%3gfhdi%4f fdsakfh+ew%4gfhdi%4f fwe8^ru&26234jä xb3#ru786234jä Recombine fdsakfh+ew%4gfhdi%4f xb3#ru786234jä xb3#ru7%4gfhdi%4f

44 Selection and Ranking if (angle = 47 power = 532) { } fdsakfh+ew%4gfhdi%4f angle = 31 xb3#ru786234jä angle = 48 xb3#ru7%4gfhdi%4f angle = 65

45 Selection and Ranking if (angle = 47 power = 532) { } angle = 51 xb4%ru786234jä xb3#ru78^^&1jä xb3#ru786234jä angle = 48 angle = 47

46 int main() { } CC Tests XML Tests GUI Tests Web Tests

47 int main() { } CC Tests XML Tests GUI Tests Web Tests

48 Testing a C Compiler CCFuzz C Source Code Compiler

49 Parsing C Files expression = n = mult(x, 2); variable call n mult x 2

50 Mutation expression = n = div(x, 2); variable call n mult div x 2

51 Recombination z = y << 3; n = mult(x, 2); expression expression = = variable << variable call z y 3 n mult x 2

52 Recombination z = y << 3; n = mult(x, 2); expression expression = = variable << variable call z y 3 n mult x 2

53 Recombination z = 2; n = mult(x, y << 3); expression expression = = variable 2 variable call z n mult x << y 3

54 Testing a C Compiler CCFuzz C Source Code Compiler

55 Evolved C Input int main() { double var0 = ; double var1 = var0; var1 /= (var * var0) L; unsigned long var2 = UL; var2 *= var2 * var2; int var3 = ; for (; var3 <= ; ++var3) { var1 = var1 / LL / (var3 & ULL) == / U; unsigned long long var4 = ULL; int var5 = ; var5 += (4 / var4 % var2 & ULL) >= / % 120; } int var4 = ; var4 = var ; signed char var5 = 22; unsigned char var6 = 7; var5 = var5 + var6; }

56 Domain Code CCFuzz C Source Code Compiler

57 int main() { } CC Tests XML Tests GUI Tests Web Tests

58 int main() { } CC Tests XML Tests GUI Tests Web Tests

59 <math> <mrow> <munderover> <mo> </mo> <mrow> <mi>x</mi> <mo>=</mo> <mn>1</mn> </mrow> <mi>n</mi> </munderover> <mrow> <apply> <power/> <ci>x</ci> <cn>2</cn> </apply> </mrow> </mrow> </math> XML Tests nx x=1 x 2

60 <math> <mrow> <munderover> <mo> </mo> <mrow> <mi>x</mi> <mo>=</mo> <mn>1</mn> </mrow> <mi>n</mi> </munderover> <mrow> <apply> <power/> <ci>x</ci> <cn>2</cn> </apply> </mrow> </mrow> </math> XML Tests

61 Domain Inputs XMLTest XML Input Tested System

62 int main() { } CC Tests XML Tests GUI Tests Web Tests

63 int main() { } CC Tests XML Tests GUI Tests Web Tests

64 EXSYST Florian Groß, Andreas Zeller

65 DROIDMATE Konrad Jamrozik, Andreas Zeller

66 DROIDMATE

67 Test Coverage Unit Test Generators GUI Test Generators Randoop Evosuite GUItar Exsyst 100 % 75 % 50 % 25 % 0 % Addressbook Calculator TerpPresent TerpSpreadSheet TerpWord

68

69

70 int main() { } CC Tests XML Tests GUI Tests Web Tests

71 int main() { } CC Tests XML Tests GUI Tests Web Tests

72

73 WEBMATE Martin Burger, Valentin Dallmeier, Andreas Zeller

74

75

76 Test this! Test Subscribe

77 int main() { } CC Tests XML Tests GUI Tests Web Tests

78 int main() { } CC Tests XML Tests GUI Tests Web Tests

79 Checking Results Crashes Assertions Regressions

80 Crashes If the system detects an error, we do so, too

81 Assertions

82 Assertions class RedBlackTree { boolean repok() { assert (roothasnoparent()); assert (rootisblack()); assert (rednodeshaveonlyblackchildren()); assert (equalnumberofblacknodesonsubtrees()); assert (treeisacyclic()); assert (parentsareconsistent());! return true; } } Test generators aim for branch coverage and thus aim for falsifying assertions

83 Regression nx n X x 2 x 3 x=1 x=1 Compare with earlier or alternate version

84 Compare Compilers LLVM Executable 0 CCFuzz GCC Executable 1 Intel Executable 0

85 App Mining

86 Layer Oracles System interface (XML, etc.) Java code C code

87 Layer Oracles System interface (XML, etc.) in = <x>20</x> Java code sx = 20 int f(int x) { return 2 * x; } void h(int x, int y) { if (x!= y && f(x) == x + 10) abort(); x = 20, y = 1 x = 20, y!= x }

88 Layer Oracles System interface (XML, etc.) Java code C code

89

90 90

91 A Rau V Dallmeier S Just E May F Gross K Herzig C Hammacher A Zeller A Tarasevich M Burger K Streit M Höschele B Pohl M Mirold I Tavecchia G Reibold J Rößler K Jamrozik JP Galeotti K Becker A Gorla

92

93 Software is manifold but can these ever be invoked from here? 1,000 Testers in a Shoe Box static analysis Advances in Automated may Test detect Generation errors here Evolutionary Algorithms int main() { } Create population Create mutations CC Tests XML Tests Rank Select Recombine GUI Tests Web Tests