Achievements and Challenges

Transcription

1 Improving Automation in Developer Testing: Achievements and Challenges Tao Xie Department of Computer Science North Carolina State University An up-to-date version of the slides is available at:

2 Who Am I? Tao Xie, Assistant Professor (2005-), Department of Computer Science, North Carolina State Univ. Received Ph.D. from Univ. of Washington Work on Automated Software Testing + Mining Software Engineering Data Automated Testing of Traditional Programs (especially OO programs) Software Security (including Security Policies) Aspect-Oriented Programs Web Services and Applications Software Designs 2

3 Why Am I Here? My research closely related to testing practices Focus on real problems Develop practical solutions Collaborate with industry Main objectives of the presentation Communicate research achievements/challenges Discuss promising solutions to address challenges Explore common ways to compare/assess tools Provide a basis for academia-industry collaborations 3

4 Why Automate Testing? Software testing is important Software errors cost the U.S. economy about $59.5 billion each year (0.6% of the GDP) [NIST 02] Improving testing infrastructure could save 1/3 cost [NIST 02] Software testing is costly Account for even half the total cost of software development [Beizer 90] Automated testing reduces manual testing effort Test execution: JUnit framework Test generation: AgitarOne, Parasoft Jtest, etc. Test-behavior checking: AgitarOne, Parasoft Jtest, etc. 4

5 Automation in Developer Testing Background on developer testing Kent Beck s 2004 talk on Future of Developer Testing This talk focuses on developer testing Not system testing etc. conducted by testers This talk focuses on automation in developer testing Not tester training/education, test process, test management, etc. This talk focuses on both research and industrial test automation tools 5

6 Software Testing Setup Test inputs + Program Outputs? = Expected Outputs Test Oracles 6

7 Software Testing Problems Test inputs + Program Outputs? = Expected Outputs Test Oracles Faster: How can tools help developers create and run tests faster? 7

8 Software Testing Problems Test inputs + Program Outputs? = Expected Outputs Test Oracles Faster: How can tools help developers create and run tests faster? Better Test Inputs: How can tools help generate new better test inputs? 8

9 Software Testing Problems Test inputs + Program Outputs? = Expected Outputs Test Oracles Faster: How can tools help developers create and run tests faster? Better Test Inputs: How can tools help generate new better test inputs? Better Test Oracles: How can tools help generate better test oracles? 9

10 Example Unit Test Case Test inputs + Program Outputs? = Expected Outputs Test Oracles void addtest() { ArrayList a = new ArrayList(1); Object o = new Object(); a.add(o); AssertTrue(a.get(0) == o); } Test Case = Test Input + Test Oracle Appropriate method sequence Appropriate primitive argument values Appropriate assertions 10

11 Faster Test inputs + Program Outputs? = Expected Outputs Test Oracles Create test inputs faster Run test inputs faster 11

12 Faster with Capture/Replay Originally used in GUI/Web testing E.g., IBM Rational Robot/Functional Tester, Mercury WinRunner, Seapine QA Wizard, AutomatedQA TestComplete, Segue SilkTest. Capture and replay the interactions between a program unit in a system and the rest of the system Capture: run a system manually or with system tests and capture interactions to the unit Replay: run unit tests created from the interactions 12

13 Capture/Replay - example System code:... ArrayList a = new ArrayList(1); HashMap h = new HashMap(); h.add(a); Object o = new Object(); a.add(o);... public void test1() { ArrayList a = new ArrayList(1); Object o = new Object(); a.add(o); } 13

14 Capture/Replay: benefits & limitations Create test inputs faster Otherwise, need manually write these unit tests Run test inputs faster Run only less-expensive unit tests (not the whole system tests) with the same effects w.r.t. the unit Limitations No new program behavior is exercised (e.g., no new bugs/coverage can be exercised beyond the whole system tests) Useful primarily for regression testing 14

15 Capture/Replay: example tools SilverMark Test Mentor Object Interaction Recording Limited on only methods with primitive arguments Parasoft Jtest Test Case Sniffer/Jtest Tracer Research Prototypes Orso&Kennedy WODA 05, Saff et al. ASE 05, Elbaum et al. FSE 06 15

16 Capture/Replay: challenges Non-primitive argument value in unit interface (De)serialization: too sensitive to implementation details Method sequence: need instrument many call sites Expensive method calls on objects (including argument objects) inside the unit E.g., DB interactions, Eclipse plugin interactions Solution: automatic mock object creation Capture/replay [Saff&Ernst PASTE 04][Parasoft Jtest]: too sensitive to changes on behavior of these method calls Upon needed by usage logics of API rets [Tillmann&Schulte ASE 06] at Microsoft Research Pex (different purpose) 16

17 Stub Creation in Parasoft Jtest Stubs are replacements for references to external methods Related to mock object creation Used e.g., when a class that uses interfaces for which an implementation has not yet been written When Jtest automatically generates test inputs, it adds stubs to test cases if a class references external resources. Allow user-defined stubs (Jtest creates templates) 17

18 Assisting Developers to Write Tests (Faster) Motivation: There exist IDE supports for developers to write production code faster but few for developers to write test code faster Example prototype: UnitPlus at NCSU Keep track of historical test code Analyze the unit code under test Recommend historical method sequences to choose for non-primitive arguments Recommend assertions to choose for asserting method behavior (more details later) [Song, Thummalapenta, and Xie ETX 07] 18

19 Software Testing Problems Test inputs + Program Outputs? = Expected Outputs Test Oracles Faster: How can tools help developers create and run tests faster? Better Test Inputs: How can tools help generate new better test inputs? Method-argument generation Method-sequence generation Better Test Oracles: How can tools help generate better test oracles? 19

20 Better Test Inputs Objectives are to make the unit execution crash (throwing uncaught exception) if possible most tools report crashes violate properties if possible cover all coverable entities (e.g., statements, blocks, and branches) most tools report code coverage Existing test inputs may not be sufficient Recall that capture/replay techniques don t go beyond the behavior exercised by existing system tests Need generate new test inputs to achieve objectives 20

21 Deal with Illegal Test Inputs Design by Contract (e.g., Parasoft Jtest, CodePro AnalytiX, MSR Spec Explorer) Define preconditions Tools generate only test inputs that satisfy preconditions Defensive Programming (e.g., AgitatorOne) Write illegal-input handling code in the beginning of the method body Tools may generate test inputs that violate preconditions 21

22 Challenges in Test-Input Generation Method-argument generation Covering branches involving complex logics in code require sophisticated argument values Need to deal with non-primitive arguments Receiver-object-state generation Similar to non-primitive-argument generation Covering branches involving complex logics in code require sophisticated receiver-object states (field values) 22

23 Method-Argument Generation Default values for specific types E.g., -1, 0, 1 for int Random values Heuristics E.g., based on values at switch, conditionals Constraint solving Example tool: Avaya Labs Research exvantage Symbolic execution [King CACM 76] Example tools: Parasoft Jtest, AgitarOne, MSR Pex 23

24 Symbolic Execution: example Adapted from slide by D. Rietmann Adapted from slide by D. Rietmann 24

25 Method-Sequence Generation What if method arguments are objects? Most tools use null or one constructor call Often need method sequences to produce desirable argument object states Similarly need method sequences to produce desirable receiver-object states of the class under test 25

26 Method-Sequence Generation cont. Random sequences within length bound Most tools use method-call sequence of length 1 i.e., constructor call + one public method call + some observer calls Parasoft Jtest 4.5 allows up to length 3 but disable this configuration in later releases (which generate tests often with length 1 by default) Sequences generated by heuristics (MSR Pex) Constructors Methods (modify this, modify any other formal parameters, or return a new result) 26

27 Method-Sequence Generation cont. Bounded exhaustive sequences Parasoft Jtest 4.5 considered as quasi-boundedexhaustive sequences Bounded exhaustive sequences by pruning equivalent concrete states [Xie et al. ASE 04][Visser et al. ISSTA 04] Bounded exhaustive sequences by pruning equivalent/subsumed symbolic states [Xie et al. TACAS 05] 27

28 Limitations of Automation Machines may not be good at guessing/generating some types of test inputs Need collaboration from developers Implicit: exploiting existing tests written by developers Explicit: allowing developers to contribute Operational profiles may be helpful [Woit ISSTA 03][Musa 04][Binder SoftwareTech 04] 28

29 Exploiting Existing Tests Manually written test code (call sites in the rest of the system) incorporates valuable domain/application knowledge Worthwhile to exploit in automated test generation Tradeoff between two extremes Exactly capture and replay in test generation Totally ignore existing tests in test generation Exploiting existing tests to help reuse/adapt existing method arguments reuse/adapt existing method sequences 29

30 Exploiting Existing Tests cont. Reuse method arguments Rostra [Xie et al. ASE 04] Reuse method sequence skeletons MSR Pex [Tillmann&Schulte IEEE Software 06] Generalize program states Jov [Xie&Notkin ASE 03], Eclat [Pacheco&Ernst ECOOP 05] Generalize method sequences/program states Substra [Yuan&Xie AST 06] 30

31 Allowing Developers to Contribute Allow to specify data factory for a class type E.g., test-input factories/test helpers in AgitarOne E.g., object repositories in Parasoft Jtest E.g., factory classes in CodePro AnalytiX and Pex Suggest quick fixes E.g., add precondition in Parasoft Jtest E.g., add defensive-programming check in Pex Allow to edit generated tests E.g., CodePro AnalytiX, Parasoft Jtest 31

32 Test Editing in CodePro AnalytiX Image from 32

33 Software Testing Problems Test inputs + Program Outputs? = Expected Outputs Test Oracles Faster: How can tools help developers create and run tests faster? Better Test Inputs: How can tools help generate new better test inputs? Better Test Oracles: How can tools help generate better test oracles? Assert behavior of individual test inputs Assert behavior of multiple test inputs 33

34 Levels of Test Oracles Expected output for an individual test input In the form of assertions in test code Properties applicable for multiple test inputs Crash (uncaught exceptions) or not, related to robustness issues, supported by most tools Properties in production code: Design by Contract (precondition, postcondition, class invariants) supported by Parasoft Jtest, CodePro AnalytiX Properties in test code: parameterized unit tests supported by MSR Pex, AgitarOne 34

35 Economics of Test Oracles Expected output for an individual test input Easy to manually verify for one test input Expensive/infeasible to verify for many test inputs Limited benefits: only for one test input Properties applicable for multiple test inputs Not easy to write (need abstraction skills) But once written, broad benefits for multiple test inputs 35

36 Assert behavior of individual test inputs Capture and Assert Capture and Assert [Xie ECOOP 06] Phase I: Capture the return values of observer methods Phase II: Create assertions to assert return values Also assert uncaught exception throwing Example tools: Parasoft Jtest, CodePro AnalytiX, AgitarOne 36

37 Capture and Assert: example public void test1() { Stack s1 = new Stack(); s1.push(3); s1.top(); s1.ismember(3); } public void test1() { Stack s1 = new Stack(); s1.push(3); assertequals(s1.top(), 3); assertequals(s1.ismember(3), true); } 37

38 Capture and Assert: issues For regression testing only unless you verify If your current version has a bug, the assertion makes sure your bug exists in a future version! Ask developers to verify assertions/exceptions in test code Example tools: Parasoft Jtest and CodePro AnalytiX 38

39 Verify Outcomes in Parasoft Jtest Image from 39

40 Capture and Assert: issues For regression testing only unless you verify If your current version has a bug, the assertion makes sure your bug exists in a future version! Ask developer to verify assertions/exceptions in test code Challenge: which observer methods to invoke? (i.e., which assertions to add?) 40

41 Recommend Assertions to Developers [Song, Thummalapenta, and Xie ETX 07] [Song, Thummalapenta, and Xie ETX 07] 41

42 Assert behavior of multiple test inputs Design by Contract Example tools: Parasoft Jtest, CodePro AnalytiX, Microsoft Research Spec Explorer, MSR Pex Class invariant: properties being satisfied by an object (in a consistent state) [AgitarOne allows a class invariant helper method used as test oracles] Precondition: conditions to be satisfied (on receiver object and arguments) before a method can be invoked Postcondition: properties being satisfied (on receiver object and return) after the method has returned Other types of specs also exist 42

43 Microsoft Research Spec Explorer - Slide adapted from MSR FSE Group

44 Assert behavior of multiple test inputs Parameterized Unit Tests Adding parameters turns unit tests into general specs [TestMethod] void AddParameterizedTest(ArrayList a, object o) { Assume.IsTrue(a!= null); int len = a.count; a.add(o); Assert.IsTrue(a[len] == o); } Read as forall a, o: the given assertion holds Tool chooses argument values that cover all implementation paths // 1. case: existing storage used AddParameterizedTest(new ArrayList(1), new object()) // 2. case: new storage allocated AddParameterizedTest(new ArrayList(0), new object()); Supported by MSR Pex [Tillmann&Schulte FSE 05] and AgitarOne - Slide adapted from MSR FSE Group 44

45 Parameterized Unit Tests in Pex Image from 45

46 Assert behavior of multiple test inputs Software Agitation in AgitarOne Software Agitation Code Observations on code behavior, plus Test Coverage data If an Observation reveals a bug, fix it Code Compile Agitate If it describes desired behavior, click to create a Test Assertion Review - Slide adapted from Agitar Software Inc. 46

47 Software Agitation in AgitarOne Image from 47

48 Test Selection/Abstraction Test selection based on code coverage, e.g., Parasoft Jtest new behavior [Xie&Notkin ASE 03, d'amorim et al. ASE 06] special behavior [Xie&Notkin ISSRE 05] Test abstraction for overall behavior [Xie&Notkin ICFEM 04] r(a(s, e).state).state == a (r(s).state, e).state 48

49 Conclusion Software testing is important and yet costly; needs automation Faster: help developers create and run tests faster Capture/replay techniques IDE supports for writing test code Better Test Inputs: help generate new better test inputs Generate method arguments Generate method sequences Better Test Oracles: help generate better test oracles Assert behavior of individual test inputs Assert behavior of multiple test inputs 49

50 Example Industrial Developer Testing Tools Agitar AgitatorOne Parasoft Jtest CodePro AnalytiX SilverMark Test Mentor Microsoft Research Pex (for.net) Microsoft Research Spec Explorer (for.net) Avaya Labs Research exvantage 50

51 ICST 2008 First International Conference on Software Testing, Verification and Validation Lillehammer, Norway April 9-11, 2008 Call for Participation Conference program includes Research papers Industry papers Student papers Various workshops

52 Questions? Contact info: Tao Xie Automated Software Engineering Group North Carolina State University Acknowledgment of feedback especially from Dave Weiss