MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler

Transcription

1 MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler René Just 1, Franz Schweiggert 1, and Gregory M. Kapfhammer 2 1 Ulm University, Germany 2 Allegheny College, USA 26th International Conference on Automated Software Engineering Lawrence, Kansas, USA November 6-12, 2011

2 Overview of MAJOR A Tool for Mutation Analysis

3 Overview of MAJOR Compiler- Integrated A Tool for Mutation Analysis

4 Overview of MAJOR Compiler- Integrated A Tool for Mutation Analysis Fast and Scalable

5 Overview of MAJOR Compiler- Integrated Domain Specific Language A Tool for Mutation Analysis Fast and Scalable

6 Overview of MAJOR Compiler- Integrated Domain Specific Language A Tool for Mutation Analysis Fast and Scalable Configurable and Extensible

7 Overview of MAJOR Compiler- Integrated Mutation Coverage Information Domain Specific Language A Tool for Mutation Analysis Fast and Scalable Configurable and Extensible

8 Overview of MAJOR Compiler- Integrated Mutation Coverage Information Domain Specific Language A Tool for Mutation Analysis Fast and Scalable Enables Optimized Workflow Configurable and Extensible

9 Overview of Mutation Analysis Mutation Analysis

10 Overview of Mutation Analysis Methodically inject small syntactical faults into the program under test Mutation Analysis

11 Overview of Mutation Analysis Methodically inject small syntactical faults into the program under test Mutation Analysis Unbiased and powerful method for assessing test oracles and input values

12 Overview of Mutation Analysis Methodically inject small syntactical faults into the program under test Mutation Analysis Unbiased and powerful method for assessing test oracles and input values Useful method for fault seeding during the empirical study of testing techniques

13 Overview of Mutation Analysis public int eval(int x){ int a=3, b=1, y; y = a * x; y += b; return y; public int max(int a, int b){ int max = a; if(b>a){ max=b; return max;

14 Overview of Mutation Analysis public int eval(int x){ int a=3, b=1, y; y = a * x; y += b; return y; public int max(int a, int b){ int max = a; if(b>a){ max=b; return max;

15 Overview of Mutation Analysis public int eval(int x){ int a=3, b=1, y; y = a * x; y += b; return y; public int max(int a, int b){ int max = a; if(b>a){ max=b; = = y = a - x; y = a + x; y = a / x; if(b < a) if(b!= a) if(b == a) return max;

16 MAJOR s Compiler MAJOR s Compiler

17 MAJOR s Compiler MAJOR s Compiler Enhanced Standard Java Compiler

18 MAJOR s Compiler Source Files MAJOR s Compiler Enhanced Standard Java Compiler

19 MAJOR s Compiler Common Compiler Options Source Files MAJOR s Compiler Enhanced Standard Java Compiler

20 MAJOR s Compiler Common Compiler Options Domain Specific Language Source Files MAJOR s Compiler Enhanced Standard Java Compiler

21 MAJOR s Compiler Common Compiler Options Domain Specific Language Source Files MAJOR s Compiler Bytecode with Embedded Mutants Enhanced Standard Java Compiler

22 MAJOR s Domain Specific Language // variable declaration listcor={&&,, ==,!=; // Define replacement list BIN(+)<"org"> -> {-,*; BIN(*)<"org"> -> {/,%; // Define own operator myop{ BIN(&&) -> listcor; BIN( ) -> listcor; COR; LVR; // Enable built-in operator AOR AOR<"org">; // Enable operator myop myop<"java.lang.system@println">;

23 MAJOR s Domain Specific Language // variable declaration listcor={&&,, ==,!=; // Define replacement list BIN(+)<"org"> -> {-,*; BIN(*)<"org"> -> {/,%; // Define own operator myop{ BIN(&&) -> listcor; BIN( ) -> listcor; COR; LVR; // Enable built-in operator AOR AOR<"org">; // Enable operator myop myop<"java.lang.system@println">; Specify mutation operators in detail

24 MAJOR s Domain Specific Language // variable declaration listcor={&&,, ==,!=; // Define replacement list BIN(+)<"org"> -> {-,*; BIN(*)<"org"> -> {/,%; // Define own operator myop{ BIN(&&) -> listcor; BIN( ) -> listcor; COR; LVR; // Enable built-in operator AOR AOR<"org">; // Enable operator myop myop<"java.lang.system@println">; Specify mutation operators in detail Define own mutation operator groups

25 MAJOR s Domain Specific Language // variable declaration listcor={&&,, ==,!=; // Define replacement list BIN(+)<"org"> -> {-,*; BIN(*)<"org"> -> {/,%; // Define own operator myop{ BIN(&&) -> listcor; BIN( ) -> listcor; COR; LVR; // Enable built-in operator AOR AOR<"org">; // Enable operator myop myop<"java.lang.system@println">; Specify mutation operators in detail Define own mutation operator groups Enable operators for a specific package, class, or method

26 Optimized Mutation Analysis Process 1 Embed and compile all mutants 2 Run test suite on instrumented program 3 Sort tests according to their runtime 4 Perform mutation analysis with reordered test suite

27 Optimized Mutation Analysis Process 1 Embed and compile all mutants 2 Run test suite on instrumented program 3 Sort tests according to their runtime 4 Perform mutation analysis with reordered test suite

28 Optimized Mutation Analysis Process 1 Embed and compile all mutants 2 Run test suite on instrumented program 3 Sort tests according to their runtime 4 Perform mutation analysis with reordered test suite

29 Optimized Mutation Analysis Process 1 Embed and compile all mutants 2 Run test suite on instrumented program 3 Sort tests according to their runtime 4 Perform mutation analysis with reordered test suite

30 Conclusion Key Concepts and Features: Compiler-integrated solution Furnishes its own domain specific language Provides mutation coverage information

31 Conclusion Key Concepts and Features: Compiler-integrated solution Furnishes its own domain specific language Provides mutation coverage information Characteristics of MAJOR: Fast and scalable technique Configurable and extensible mutation tool Enables an optimized workflow for mutation analysis

32 Figure: Multiple mutated binary expression as the right hand side of an assignment statement. public int eval(int x){ int a = 3, b = 1, y; y = (M_NO==1)? a - x: (M_NO==2)? a + x: (M_NO==3)? a % x: (M_NO==0 && COVERED(1,3))? a * x : a * x; // original if(m_no==4){ y -= b; else if(m_no==0 && COVERED(4,4)){ y += b; else{ y += b; // original return y; Figure: Collecting coverage information. Figure: Integration of the conditional mutation approach into the compilation process. listcor={&&,, ==,!=; method ="java.lang.system@println"; myop{ BIN(&&) -> listcor; BIN( ) -> listcor; COR; LVR; // Define replacement list BIN(+)<"org"> -> {-,*; BIN(*)<"org"> -> {/,%; // Enable built-in operator AOR AOR<"org">; // Enable operator myop myop<method>; Figure: DSL script to define the mutation process. package major.mutation; public class Driver{ public static int MAX_NO = ; public static int[] COV = new int[max_no]; public static int M_NO=0; public static boolean COVERED (int from, int to){ for(int i=from; i<=to; ++i){ COV[i]++; return false; Figure: Simple driver class implementation. Figure: Minimizing the runtime of mutation analysis by means of test prioritization and mutation coverage. Figure: Compiler runtime to generate and compile the mutants for all of the projects. Figure: Runtime of the mutation analysis processes. Do you want to learn more details about MAJOR? MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler René Just 1 and Franz Schweiggert 1 and Gregory M. Kapfhammer 2 1 Department of Applied Information Processing, Ulm University 2 Department of Computer Science, Allegheny College IMPORTANT CONTRIBUTIONS Enhances the Java 6 Standard Edition compiler Provides its own domain specific language (DSL) Easily applicable in all Java development environments Effectively reduces mutant generation time to a minimum CONDITIONAL MUTATION Transforms the program s abstract syntax tree (AST) Encapsulates the mutations within conditional statements ASSIGN IDENT COND-EXPR y ASSIGN THEN COND ELSE IDENT BINARY BINARY (M NO ==1) COND-EXPR y * - a x a x THEN COND ELSE BINARY (M NO ==2) BINARY + * a x a x SUPPORTED FEATURES Simple compiler options enable the mutation analysis Configurable mutation operators by means of a DSL Determination of mutation coverage by running the original code MUTATION COVERAGE It is impossible to kill a mutant if it is not reached and executed Additional instrumentation determines the covered mutations Mutation coverage is only examined if the tests execute the original code An external driver efficiently records the covered mutations as ranges Only those mutants covered by a test case are executed IMPLEMENTATION DETAILS OPTIMIZED MUTATION ANALYSIS PROCESS 1. Embed and compile all mutants into the original program 2. Run tests on original program to gather runtime and coverage 3. Sort tests in ascending order according to their runtime 4. Perform mutation analysis while employing the reordered test suite and the mutation coverage information Compiler runtime in seconds Number of killed mutants RUNTIME OF MAJOR S COMPILER Number of mutants apache ant jfreechart itext java pathfinder commons math commons lang numerics4j Negligible overhead for generating and compiling the mutants Applicable on commodity workstations, even for large projects EVALUATION OF MUTATION ANALYSIS PROCESSES optimized order (using coverage information) 40 random order (using coverage information) original order (using coverage information) optimized order (without coverage information) 20 random order (without coverage information) original order (without coverage information) FUTURE WORK Runtime in seconds Implementing new mutation operators at the semantic level Extending the domain specific language to support new operators Integrating conditional mutation into the new Java 7 compiler rene.just@uni-ulm.de 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011) gkapfham@allegheny.edu See you tomorrow for a live demonstration!