EQUIVALENCE PARTITIONING AS A BASIS FOR DYNAMIC CONDITIONAL INVARIANT DETECTION

Transcription

1 EQUIVALENCE PARTITIONING AS A BASIS FOR DYNAMIC CONDITIONAL INVARIANT DETECTION Worakarn Isaratham Supervised by Dr Rosemary Monahan July 2015

2 OUTLINE 1. Motivation 2. Background 3. Solution 4. Evaluation 5. Conclusion Worakarn Isaratham 15 June 2015

3 MOTIVATION Worakarn Isaratham 3 15 June 2015

4 Formal Specification Test Suite Software Worakarn Isaratham 1 15 June 2015

5 Detect formal specification by Formal Specification Test Suite observing dynamic test run Software Worakarn Isaratham 1 15 June 2015

6 HOW CAN EQUIVALENCE PARTITIONING ASSIST DYNAMIC DETECTION OF CONDITIONAL INVARIANTS? Worakarn Isaratham 2 15 June 2015

7 BACKGROUND Worakarn Isaratham 7 15 June 2015

8 EQUIVALENCE PARTITIONING Divide input/output domains into equivalence classes, conditional to how data are processed All members of a class are processed in the same way, but different from other classes Worakarn Isaratham 3 15 June 2015

9 DYNAMIC INVARIANT DETECTION Invariants properties of programs that hold true for all executions Dynamic invariants detection detect invariants by observing dynamic execution of the target program. Worakarn Isaratham 4 15 June 2015

10 DAIKON INVARIANT DETECTOR Worakarn Isaratham 5 15 June 2015

11 DAIKON INVARIANT DETECTOR Worakarn Isaratham 5 15 June 2015

12 DAIKON INVARIANT DETECTOR Worakarn Isaratham 5 15 June 2015

13 CONDITIONAL INVARIANTS Invariants in the implication form p q Infeasible to compute exhaustively under dynamic invariant detection Worakarn Isaratham 6 15 June 2015

14 SPLITTERS Worakarn Isaratham 7 15 June 2015

15 SOLUTION Worakarn Isaratham June 2015

16 YACON Worakarn Isaratham 8 15 June 2015

17 YACON Worakarn Isaratham 9 15 June 2015

18 EXTRACTION PHASE Worakarn Isaratham June 2015

19 EXTRACTION PHASE Boundary value recovery strategy Looking for boundary values in test data Test suite invariants recovery strategy Arguments passed to the same method from the same position should be in the same equivalence class Support user-defined strategies Worakarn Isaratham June 2015

20 BOUNDARY VALUE STRATEGY 1. Run Chicory (Daikon s instrumenter for Java) to collect trace data 2. Analyse the trace data for adjacent values 3. Create interval-based partitions Worakarn Isaratham June 2015

21 BOUNDARY VALUE STRATEGY Worakarn Isaratham June 2015

22 TEST SUITE INVARIANTS STRATEGY 1. Proxify the test suite (rewrite source files by creating new method for each invocation on target classes) Target t = new Target(); int i = t.foo( abc, 0); Target t = new Target(); int i = ẎȧċȯṅProxifier.proxify(t).m foo Target 42( abc, 0) Worakarn Isaratham June 2015

23 TEST SUITE INVARIANTS STRATEGY 2. Compile the proxified code 3. Run Daikon on proxified test suite 4. Transform Daikon s result into predicate-based partitions Worakarn Isaratham June 2015

24 TEST SUITE INVARIANTS STRATEGY Worakarn Isaratham June 2015

25 TRANSLATION PHASE Worakarn Isaratham June 2015

26 TRANSLATION PHASE 1. Read partitioning files 2. Convert partitions into splitting conditions Worakarn Isaratham June 2015

27 EVALUATION Worakarn Isaratham June 2015

28 EVALUATION Partitioning Recovery Effectiveness Invariants Discovery Effectiveness Performance (runtime) Worakarn Isaratham June 2015

29 RECOVERY EFFECTIVENESS Compare generated partitions against expected Measure best-matched distance of each domain d(c 1,C 2 )= n i=1 min δ(s i,s j=1..m j)+ Worakarn Isaratham June 2015 m j=1 min i=1..n δ(s j,s i ) 0, S i = S j δ(s i,s j )= w s, S i S j S j S i 1, 1

30 RECOVERY EFFECTIVENESS Calculate overall effectiveness of the generated partitions =1 2 ( 1 1 Worakarn Isaratham June 2015

31 RECOVERY EFFECTIVENESS Worakarn Isaratham June 2015

32 INVARIANTS DISCOVERY EFFECTIVENESS Measure the effect of using Yacon by comparing generated invariants from (1) Daikon (2) Daikon+Yacon (3) Daikon+Expected Partition Measure quality metrics Worakarn Isaratham June 2015

33 INVARIANTS DISCOVERY EFFECTIVENESS Assess quality of each invariant Correctness is it true for all conceivable inputs? Usefulness can it help programmers in some ways? Relevance is it a characteristic of this program? Worakarn Isaratham June 2015

34 INVARIANTS DISCOVERY EFFECTIVENESS Assess quality of partitioning Correctness correct invariants / reported invariants Usefulness useful invariants / reported invariants Precision relevant invariants / reported invariants Recall relevant invariants / expectation Worakarn Isaratham June 2015

35 INVARIANTS DISCOVERY EFFECTIVENESS Precision Worakarn Isaratham June 2015

36 INVARIANTS DISCOVERY EFFECTIVENESS Recall Worakarn Isaratham June 2015

37 INVARIANTS DISCOVERY EFFECTIVENESS Worakarn Isaratham June 2015

38 PERFORMANCE Worakarn Isaratham June 2015

39 THREATS TO VALIDITY Small sample size 7 programs, 11 test suites Small programs textbook programs less than 250 lines of code Selection bias some programs are selected because they have desired characteristics Subjectivity of invariants assessment Test suite construction bias test suites written to fit how Yacon works Worakarn Isaratham June 2015

40 CONCLUSION Worakarn Isaratham June 2015

41 CONCLUSION Information from equivalence partitioning can be effective in uncovering conditional invariants. Our recovery strategies work better together than individually. The recovery strategies are only effective in limited circumstances The solution increases recall metric, at the expense of overall quality of generated invariants Worakarn Isaratham June 2015

42 FUTURE WORK Find more effective partitioning recovery strategies. Automatic invariants assessment to overcome the subjectivity in invariants evaluation and to work at larger scale. Compare Yacon against other ways of generating splitting conditions. Adapt Yacon to other invariant detectors. Worakarn Isaratham June 2015

43 THANK YOU 43

44 44

45 (BACKUP)

46 EQUIVALENCE PARTITIONING [it] is a technique that is intuitively used by virtually every tester we've ever met. Rick D Craig and Stefan P Jaskiel. Systematic software testing. Artech House, 2002

47 INVARIANTS Includes Class invariants Method preconditions Method postconditions NOT INCLUDE LOOP INVARIANTS

48 SPLITTING POLICIES Default Procedure Return Analysis Shipped with Daikon Static Analysis (for IF, FOR, WHILE statements) Cluster Analysis Random Sampling

49 PROGRAM SIZE 49

50 PROGRAM SELECTION All are from textbooks StackAr, QueueAr are stack and queue data structures, often used as benchmark programs for Daikon BinaryHeap, BinarySearchTree Built-in test suites Worakarn Isaratham June 2015

51 PROGRAM SELECTION Earthquake, ComputeTax, Insurance Suitable structure for equivalence partitioning No built-in test suites Worakarn Isaratham June 2015

52 TESTING Worakarn Isaratham June 2015

53 Yacon Daikon Celeriac 53 Chicory