Implement detection of duplicate test coverage for Java applications

Transcription

1 Masaryk University Faculty of Informatics Implement detection of duplicate test coverage for Java applications Bachelor s Thesis Jakub Schwan Brno, Spring 2018

2

3 Replace this page with a copy of the official signed thesis assignment and a copy of the Statement of an Author.

4

5 Declaration Hereby I declare that this paper is my original authorial work, which I have worked out on my own. All sources, references, and literature used or excerpted during elaboration of this work are properly cited and listed in complete reference to the due source. Jakub Schwan Advisor: Mgr. Marek Grác, Ph.D. i

6

7 Acknowledgements I would like to express my gratitude to my supervisor Mgr. Marek Grác, Ph.D., and consultant Ing. Radovan Synek for their willingness, guidance, valuable advice and comments, and time invested in consultations throughout the work. Other thanks belong to the JaCoCo community, especially to Marc R. Hoffmann, for providing expert advice when analyzing JaCoCo. iii

8 Abstract The thesis aims to analyze and implement detection of duplicate tests coverage as part of the open sourcing tool for tests code coverage and to represent the results of founded duplications appropriately. The result of this work should help developers and testers to optimize their test runs. iv

9 Keywords Java, testing, code coverage, duplicate test detection, JaCoCo, bytecode, ASM v

10

11 Contents Introduction 1 1 Code coverage in the Java ecosystem Java Java bytecode Code coverage Statement coverage Branch coverage ASM library Class visitors Method visitors JaCoCo Features of JaCoCo tool Control Flow Analysis for Java Methods Instrumentation of the source code Collecting of code coverage data Coverage report for the source code Design of detection duplicate code coverage Obtain a location of the call Filtering out the location of the source Replacement of the inserted bytecode Runtime of the calls Storing of the duplications Report for users with information about duplicate test coverage Extended JaCoCo with the new functionality Possible performance issues of the design Implementation Filtering method Insert adjust probe with external method call An inserted probe that calls static method An inserted probe that calls a constructor of the object Conclusion 37 vii

12 Bibliography 39 viii

13 List of Tables 1.1 A JaCoCo Flow Edges A JaCoCo probes bybtecode instructions The overhead per probe 18 ix

14

15 List of Figures 1.1 Code Coverage Report for JaCoCo Code Coverage Report for JaCoCo Detail of code coverage for a java class from JaCoCo report Filtering method for getting a location of the test call. 32 xi

16

17 Introduction Automated testing is one of the methods used to verify the proper functionality of the software. Auto-tests have many advantages over manual testing because they are fast and their management and development usually requires fewer human resources than manual testing of the application. Automated testing enables the application to be tested quickly and inexpensively, even on multiple system platforms with different specifications. Applications are written in different programming languages. One of the most used is the Java programming language. For Java-written applications, there are plenty of test libraries, such as JUnit or AssertJ or others, and can be tested thoroughly to help them. Because there are even bigger applications where it is challenging to find out what is covered by the tests and for which parts of the tests are completely missing, tools have been developed to help automate testing source code coverage. Tools that are used for Java-written applications include JaCoCo, jcov, and others. With these tools, we can implement automated tests that effectively cover almost the entire source code of the application. However, verifying such an extensive application can be very time-consuming, and runtime can be several hours, even though tests run on powerful computers. If they can not experience technical problems in a long test run, then duplicate source code coverage can be one of the causes of automated testing. For such large applications, it is not only difficult to trace such duplicates but also to decide whether it is a duplicate of the test. If the tests run every time the source code is changed, then the great emphasis is on running the tests as quickly as possible, and the time lags are undesirable. With such frequent iterations of tests, it is possible to run only some parts of tests that include a covered code, but such parts may have a long run. For a large testing suits that have a long runtime, it is necessary to have more testing machines if the application should be tested in a certain amount of time. And more testing machines, which are not always used at 100%, means higher financial costs for the testing environment. 1

18 This thesis aims to design and implement detection of duplicate coverage tests for Java applications as part of the open sourcing tool for detected pore tests and to represent the results found by duplication appropriately. As a result, developers and testers can easily detect duplicate tests and make it easier to determine whether duplicates can be deleted. Part of the test is also the recording of the test run. The reader is expected to have basic knowledge of the Java programming language, Java Virtual Machine, and knows the principles of automated application testing. 2

19 1 Code coverage in the Java ecosystem 1.1 Java Programing language Java is an object-oriented programing language developed since the year 1991 and was first released in The syntax of this programing language is based on programming languages C and C++. An application written in programing language Java standardly walks through five phases - editing, compiling, loading, verifying and executing. Java is not translated into executable code, i.e., machine source code, but into pseudocode called bytecode [1]. Application compiled into the bytecode is executed by JVM 1 regardless of computer architecture or operating system Java bytecode Java bytecode is the instruction set of the Java virtual machine. Each instruction consists of a one-byte opcode followed by zero or more operands. For example, iadd, which will receive two integers as an operand and add them together. Java Virtual Machine: To understand the details of the bytecode, we need to discuss how a Java Virtual Machine works regarding the execution of the bytecode. JVM is a platform-independent execution environment that converts Java bytecode into machine language and executes it. A JVM is a stackbased machine. Each thread has a JVM stack which stores frames. A frame is created each time a method is invoked and consists of an operand stack, an array of local variables, and a reference to the constant runtime pool of the class of the current method. [2] Stack Based Virtual Machines: We need to know a little about stack-based VM to understand Java Bytecode better. A stack-based virtual machine the memory structure 1. JVM Java Virtual Machine 3

20 1. Code coverage in the Java ecosystem where the operands are stored is a stack data structure. Operations are carried out by popping data from the stack, processing them and pushing in back the results in LIFO (Last in First Out) fashion. In a stack-based virtual machine, the operation of adding two numbers would usually be carried out in the following manner. [3] 1.2 Code coverage Code coverage is a measure which describes how many percents of the code was used with individual tests. Applications with high percent test coverage than have a much lesser tendency to display untreated error rates of the software than systems that have a small percentage of tests coverage. To determine how much of the code is covered by our tests, the coverage criteria are used, which are the rules or requirements that the test set must satisfy. There are several different coverage criteria. The main coverage criteria being [4]: Function coverage This criteria checks if each function (or subroutine) has been called in the program. Statement coverage This criteria checks if each statement has been executed. Branch coverage This criteria checks if each branch of each controller structure (e.g., if or case statements), that can be called during program execution, has been executed. For example, if we have an if statement in which are both the true and false branches, then is checked that tests for the program execute both branches. Condition coverage This criteria checks if each Boolean subexpression evaluated both to true and false conditions. Some criteria can be combined because their principles are quite similar and thus obtain a more accurate result of code coverage by tests. For example, if we associate Branch coverage and Condition coverage, we do not have to measure the coverage rate separately for these criteria, because we have a condition for each decision, and we can go one of the possible branches. 4

21 1. Code coverage in the Java ecosystem Statement coverage Statement coverage is the most basic form of code coverage. A statement is covered if it is executed. Note that a statement does not necessarily correspond to a line of code. Multiple statements on a single line can confuse issues - the reporting if nothing else. Where there are sequences of statements without branches it is not necessary to count the execution of every statement, just one will suffice, but people often like the count of every line to be reported anyway, especially in summary statistics. [5] This type of coverage is relatively weak in that even with 100% statement coverage there may still be serious problems in a program which could be discovered through the use of other metrics. Even so, the first time that statement coverage is used in any reasonably sized development effort, it is very likely to show up some bugs. It can be quite challenging to achieve 100% statement coverage. There may be sections of code designed to deal with error conditions or rarely occurring events such as a signal received during a particular segment of the code. There may also be code that should never be executed: if ( $param > 20) { die " This should never happen!"; } It can be useful to mark such code in some way and flag an error if it is executed. Statement coverage is also known as C0, statement execution, line coverage, segment coverage or basic block coverage. Basic block coverage is the same as statement coverage except for the unit of code measured is each sequence of non-branching statements Branch coverage This metric reports whether Boolean expressions tested in control structures (such as the if-statement and while-statement) evaluated to both true and false. The entire Boolean expression is considered one true-or-false predicate regardless of whether it contains logical-and or logical-or operators. Additionally, this metric includes coverage of 5

22 1. Code coverage in the Java ecosystem switch-statement cases, exception handlers, and all points of entry and exit. Constant expressions controlling the flow are ignored. [5] The goal of branch coverage is to ensure that whenever a program can jump, it jumps to all possible destinations. The most simple example is complete if statement: if ($x) { print } else { print } "a"; "b"; Full coverage is only achieved here only if $x is true on one occasion and false on another. Achieving full branch coverage will protect against errors in which some requirements are not met in a specific branch. For example: if ($x) { $h = { a => 1 } } else { $h = 0; } print $h ->{a}; This code will fail if $x is false (and you are using strict refs). In such a simple example statement coverage is as powerful, but branch coverage should also allow for the case where the else part is missing, and in languages which support the construct, switch statements should be catered for: $h = 0; if ($x) { $h = { a => 1 } } print $h ->{a}; 100% branch coverage implies 100% statement coverage. [5] Branch coverage is also known as C1, decision coverage or all-edges coverage. 6

23 1. Code coverage in the Java ecosystem 1.3 ASM library ASM 2 is an all purpose Java bytecode manipulation and analysis framework. It can be used to modify existing classes or to generate classes, directly in binary form dynamically. ASM provides some common bytecode transformations and analysis algorithms from which custom complex transformations and code analysis tools can be built. ASM offers similar functionality as other Java bytecode frameworks but is focused on performance. Because it was designed and implemented to be as small and as fast as possible, it is well suited for use in dynamic systems (but can, of course, be used statically too, e.g., in compilers). [6] The goal of the ASM library is to generate, transform and analyze compiled Java classes, represented as byte arrays (as they are stored on disk and loaded in the Java Virtual Machine). For this purpose, ASM provides tools to read, write and transform such byte arrays by using higher level concepts than bytes, such as numeric constants, strings, Java identifiers, Java types, Java class structure elements, etc. The scope of the ASM library is strictly limited to reading, writing, transforming and analyzing classes. [7] ASM library provides a simple API 3 for decomposing, modifying, and recomposing binary Java classes. ASM exposes the internal aggregate components of a given Java class through its visitor-oriented API. It also provides, on top of this visitor API, a tree API that represents classes as object constructs. Both APIs can be used for modifying the binary bytecode, as well as generating new bytecode (via injection of new code into the existing code, or through the generation of new classes altogether). [8] Class visitors Since the visit of the class members can be interleaved (it is possible to start visiting a field, then start visiting a method, go back to visit annotations of the field, continue with some instructions of the method, visit attributes of the field, add new instructions to the method, and 2. ASM Keyword of C, which allows implements some function in assembly language. 3. API Application Programming Interface 7

24 1. Code coverage in the Java ecosystem so on), it is not possible to construct the class file s byte array in a sequential order, from beginning to end. Instead, it is necessary to use several byte vectors that can grow simultaneously. This is why there are several writer classes, unlike for the reader case. [9] ClassVisitor is an abstract class from org.objectweb.asm package which provides methods to visit a Java class. Methods from this class must be called in specific order. ClassVisitor is used as a root of the tree-base from which is possible to visit fields, methods, annotations, inner classes and so on. The ClassWriter class is the main entry point. It contains the class header elements and the lists of its fields and methods, as well as a SymbolTable instance, holding the constant pool items and the bootstrap methods of the class. This symbol table uses a hash set of Symbol objects, to avoid adding the same item several times in the constant pool or the bootstrap methods array. The symbol table can be created from an existing class by passing a ClassReader argument to its constructor. This allows unchanged methods to be copied as is from a class reader to a class writer, without visiting their content. ClassWriter extends a ClassVisitor that generates a corresponding ClassFile structure, as defined in the Java Virtual Machine Specification. ClassReader is a parser which makes a ClassVisitor visit a Class- File structure, as defined in the Java Virtual Machine Specification. This class parses the ClassFile content and calls the appropriate visit methods of a given ClassVisitor for each field, method and bytecode instruction encountered. ClassWriter can be used alone, to generate a new Java class. With ClassReader and adapter, ClassVisitor can be generated a modified class from an existing Java class. [10, 11] Method visitors Generating whole methods in the ASM API is more involved than other operations in the class. This involves a significant amount of low-level byte-code manipulation. Method visitor has implemented methods for bytecode manipulation in compiled classes. For practical uses is better to modify an existing method to make it more accessible or change a whole class to make it extensible. 8

25 1. Code coverage in the Java ecosystem For visiting Java methods is in ASM API package org.objectweb.asm class named MethodVisitor. Usage of this class is intuitive and similar to ClassVisitor, that means that methods of this class must be called in specific order. Implemented methods of the MethodVisito class required as input an integer representation of the JVM instruction. In ASM API is an interface called Opcodes, which contains constants representations of JVM instructions. This interface does not define all the JVM opcodes because some opcodes are automatically handled [12]. ClassWriter can be used alone, to generate a new Java class. With ClassReader and adapter, ClassVisitor can be generated a modified class from an existing Java class. [10, 11] Java Virtual Machine instructions A Java Virtual Machine instruction consists of an opcode specifying the operation to be performed, followed by zero or more operands embodying values to be operated. Here is a list and description of the selected JVM instructions required for further work: aload Load reference from the local variable. The index is an unsigned byte that must be an index into the local variable array of the current frame. The local variable at index must contain a reference. The objectref 4 in the local variable at index is pushed onto the operand stack. bastore Store into byte or boolean array. The arrayref 5 must be of type reference and must refer to an array whose components are of type byte or type boolean. The index and the value must both be of type int. The arrayref, index, and value are popped from the operand stack. The int value is truncated to a byte and stored as the component of the array indexed by index. iconst_<i> Push the int constant <i> (-1, 0, 1, 2, 3, 4 or 5) onto the operand stack. 4. objectref an object reference 5. arrayref an array reference 9

26 1. Code coverage in the Java ecosystem bipush Push byte. The immediate byte is sign-extended to an int value. That value is pushed onto the operand stack. new Create a new object. The unsigned indexbyte1 and indexbyte2 are used to construct an index into the run-time constant pool of the current class. The run-time constant pool item at the index must be a symbolic reference to a class or interface type. The named class or interface type is resolved and should result in a class type. Memory for a new instance of that class is allocated from the garbage-collected heap, and the instance variables of the new object are initialized to their initial default values. The objectref, a reference to the instance, is pushed onto the operand stack. invokestatic Invoke a class (static) method. The unsigned indexbyte1 and indexbyte2 are used to construct an index into the run-time constant pool of the current class. The run-time constant pool item at that index must be a symbolic reference to a method, which gives the name and descriptor of the method as well as a symbolic reference to the class in which the method is to be found. The named method is resolved. The resolved method must be static, and therefore cannot be abstract. On successful resolution of the method, the class that declared the resolved method is initialized if that class has not already been initialized. The operand stack must contain nargs argument values, where the number, type, and order of the values must be consistent with the descriptor of the resolved method. invokespecial Invoke instance method; special handling for the superclass, private, and instance initialization method invocations. The unsigned indexbyte1 and indexbyte2 are used to construct an index into the run-time constant pool of the current class. The run-time constant pool item at that index must be a symbolic reference to a method, which gives the name and descriptor of the method as well as a symbolic reference to the class in which the method is to be found. The named method is resolved. 10

27 1. Code coverage in the Java ecosystem Next, the resolved method is selected for invocation unless all of the following conditions are true: The ACC_SUPER flag is set for the current class, the class of the resolved method is a superclass of the current class, and the resolved method is not an instance initialization method. If the conditions are true, the actual method to be invoked is selected by the following lookup procedure. Let C be the direct superclass of the current class: If C contains a declaration for an instance method with the same name and descriptor as the resolved method, then this method will be invoked. The lookup procedure terminates. Otherwise, if C has a superclass, this same lookup procedure is performed recursively using the direct superclass of C. The method to be invoked is the result of the recursive invocation of this lookup procedure. Otherwise, an AbstractMethodError is raised. The objectref must be of type reference and must be followed on the operand stack by nargs argument values, where the number, type, and order of the values must be consistent with the descriptor of the selected instance method. areturn Return reference from the method. The objectref must be of type reference and must refer to an object of a type that is assignment compatible with the type represented by the return descriptor of the current method. If the present method is a synchronized method, the monitor entered or reentered on invocation of the method is updated and possibly exited as if by execution of a monitorexit instruction in the current thread. If no exception is thrown, objectref is popped from the operand stack of the current frame and pushed onto the operand stack of the frame of the invoker. Any other values on the operand stack of the present method are discarded. return Return void from the method. The current method must have return type void. If the present method is a synchronized method, the monitor entered or reentered on invocation of the method is updated and possibly exited as if by execution of a monitorexit instruction in the current thread. If no exception is thrown, any values on the operand stack of the current frame are discarded. 11

28 1. Code coverage in the Java ecosystem dup Duplicate the top value on the operand stack and push the duplicated value onto the operand stack. 1.4 JaCoCo JaCoCo 6 is an open source code coverage tool for application wrote in programing language Java [13]. JaCoCo has been created as a replacement for old and unused code coverage tool EMMA. The main goal of JaCoCo project is to provide a new standard for analyzing code coverage for applications running on JVM. JaCoCo is easy, flexible and well-documented library that can be easily used and combined with existing scripts and tools. JaCoCo has good performance when using, and in particular, for large projects, a minimum of overhead is required Features of JaCoCo tool Features of JaCoCo tool which is good to know. Code coverage criteria are used to analyze coverage for instructions (C0), branches (C1), rows, methods, variables and complexity of cycles. [14] Functionality is based on Java bytecode 7, so analysis can also be performed without source code 8. Simple Java-based integration based on on-the-fly instrumentation. It is possible to use other instrumentation scenarios that are available through the JaCoCo API. It can be easily integrated with all applications running on Java VM 9. This makes it possible to use it to analyze code coverage of plain Java programs, web containers, or EJB servers 10. Compatible with all released Java class file versions. 6. JaCoCo Java Code Coverage 7. Java bytecode is stored in files with.class extension. 8. The source code for Java is stored in files with the.java extension. 9. VM virtual machine 10. EJB Enterprise Java Beans 12

29 Support for different JVM languages. 1. Code coverage in the Java ecosystem Several report formats of analytical outputs (HTML, XML, CSV). Remote protocol and JMX 11 control to request execution data dumps from the coverage agent at any point in time. Ant tasks to collect and manage execution data and create structured coverage reports. Maven plug-in to collect coverage information and create reports in Maven builds Control Flow Analysis for Java Methods Implementing a coverage tool that supports statement (C0) as well as branch coverage (C1) requires detailed analysis of the internal control flow of Java methods. Due to the architecture of JaCoCo, this analysis happens on the bytecode of compiled class files. [15] A graph can represent the possible control flow in the bytecode. The nodes are bytecode instruction, the edged of the graph represent the possible control flow between the instructions. The example of the control flow represented by a graph is shown in the figure 1.1 at page 14. In the image, the location marked with P is the location where the probe is to be inserted. Flow Edges The control flow graph of a Java method defined by Java bytecode may have the following Edges. Each edge connects a source instruction with a target instruction. In some cases, the source instruction or the target instruction does not exist (virtual edges for method entry and exit) or cannot be exactly specified (exception handlers). In the table 1.1 at page 15 is an overview of the flow edges Instrumentation of the source code Instrumentation requires mechanisms to modify and generate Java bytecode. JaCoCo uses the ASM library for this purpose internally [16]. 11. JMX Java Management Extensions 13

30 1. Code coverage in the Java ecosystem [15] Figure 1.1: Code Coverage Report for JaCoCo 14

31 1. Code coverage in the Java ecosystem Table 1.1: A JaCoCo Flow Edges Type Source Target Remarks ENTRY - First instruction in method SEQUENCE JUMP Instruction, except GOTO, xreturn, THROW, TA- BLESWITCH and LOOKUP- SWITCH GOTO, IFx, TA- BLESWITCH or LOOKUP- SWITCH instruction EXHANDLER Any instruction in handler scope EXIT xreturn or THROW instruction Subsequent instruction Target instruction Target instruction TABLESWITCH and LOOKUP- SWITCH will define multiple edges. EXEXIT Any instructioception. - Unhandled ex- [15] - 15

32 1. Code coverage in the Java ecosystem The ASM library is lightweight, easy to use and very efficient regarding memory and CPU usage. It is actively maintained and includes as huge regression test suite. Using of the ASM library for JaCoCo is described in section 1.3 which starts at page 7. Probe Insertion Strategy Probes are additional instructions that can be inserted between existing instructions. They do not change the behavior of the method but record the fact that they have been executed. One can think probes are placed on edges of the control flow graph. Theoretically, we could insert a probe at every edge of the control flow graph. As a probe implementation itself requires multiple bytecode instructions, this would increase the size of the class files several times and significantly slow down execution speed of the instrumented classes. Fortunately this is not required, in fact, we only need a few probes per method depending on the control flow of the method. For example, a method without any branches requires a single probe only. The reason for this is that starting from a certain probe we can back-trace the execution path and typically get coverage information for multiple instructions. [15] If a probe has been executed, we know that the corresponding edge has been visited. From this edge we can conclude to other preceding nodes and edges: If an edge has been visited, we know that the source node of this edge has been executed. If a node has been executed and the node is the target of only one edge we know that this edge has been visited. Recursively applying these rules allows determining the execution status of all instructions of a method given that we have probes at the right positions. Therefore JaCoCo inserts probes 16 at every method exit (return or throws) at every edge where the target instruction is the target of more than one edge.

33 1. Code coverage in the Java ecosystem We recall that a probe is simply a small sequence of additional instructions that need to be inserted at a control flow edge. The following table illustrates how these extra instructions are added in case of different edge types. Probe Implementation Code coverage analysis is a runtime metric that provides execution details of the software under test. This requires detailed recording of the instructions (instruction coverage) that have been executed. For branch coverage also the outcome of decisions has to be recorded. In any case, execution data is collected by so-called probes. A probe is a sequence of bytecode instructions that can be inserted into a Java method. When the probe is executed, this fact is recorded and can be reported by the coverage runtime. The probe must not change the behavior of the original code. The only purpose of the probe is to record that it has been executed at least once. The probe does not record the number of times it has been called or collects any timing information. The latter is out of scope for code coverage analysis and more in the objective of a performance analysis tool. Typically multiple probes need to be inserted into each method. Therefore probes need to be identified. Also, the probe implementation and the storage mechanism it depends on needs to be thread safe as multi-threaded execution is a common scenario for java applications (albeit not for plain unit tests). Probes must not have any side effects on the original code of the method. Also, they should add minimal overhead. So to summarize the requirements for execution probes: Record execution Identification for different probes Thread safe No side effects on application code Minimal runtime overhead JaCoCo implements probes with a boolean[] array instance per class. Each probe corresponds to an entry in this array. Whenever the probe is executed the entry is set to true with the following four bytecode instructions shown in table

34 1. Code coverage in the Java ecosystem Table 1.2: A JaCoCo probes bybtecode instructions ALOAD xpush ICONST_1 BASTORE probearray probeid Table 1.3: The overhead per probe Possible Opcodes Min. Size [bytes] Max. Size [bytes] ALOAD_x, ALOAD 1 2 ICONST_x, BIPUSH, 1 3 SIPUSH, LDC, LDC_W ICONST_1 1 1 BASTORE 1 1 Total: 4 7 Note that this probe code is thread-safe, does not modify the operand stack or modify local variables and is also thread safe. It does also not leave the method through an external call. The only prerequisite is that the probe array is available as a local variable. For this, at the beginning of each method, additional instrumentation code needs to be added to obtain the array instance associated with the belonging class. To avoid code duplication, the initialization is delegated to a static private method jacocoinit() which is added to every non-interface class. The size of the probe code above depends on the position of the probe array variable, and the value of the probe identifier as different opcodes can be used. As calculated in the table below the overhead per probe ranges between 4 and 7 bytes of additional bytecode shown in table 1.3. [15] 18

35 Implementation of the probe in JaCoCo 1. Code coverage in the Java ecosystem The implementation of the probe by calling methods from the ASM library looks like this: public void insertprobe ( final int id) { mv. visitvarinsn ( Opcodes.ALOAD, variable ); InstrSupport. push (mv, id ); mv. visitinsn ( Opcodes. ICONST_1 ); mv. visitinsn ( Opcodes. BASTORE ); } MethodVisitor instance (in example mv) is already configured by JaCoCo. The input value of the method that inserts the probe into the source code is the id of the probe. The injected code must load an array that store information about which code was executed. This is done using the method visitvarinsn(). Then, using the internal method, on the stack is placed the position on which the code passes. A constant value is then entered on the stack which indicates the code passing. Finally, these changes are saved. This is done using the operation BASTORE Collecting of code coverage data Coverage data are collected and presented automatically when the application finishes running. If the application is forcibly stopped, then it is not possible to see the results of the analysis. Data about code coverage of the particular program run are stored in coverage sessions. It contains the list of considered Java classes and Java packages along with the recorded coverage details. These sessions are automatically created after the end of each run or can be returned at the moment when the user sends a request for the session. At the end of the whole coverage analyze is created a coverage report from all coverage sessions and all sessions are removed. One Coverage session usually represents a Java package. During analysis, there are several sessions mainly if tests suites are run in parallel. 19

36 1. Code coverage in the Java ecosystem JaCoCo Agent JaCoCo uses class file instrumentation to record execution coverage data. Class files are instrumented on-the-fly using a so called Java agent. This mechanism allows in-memory pre-processing of all class files during class loading independent of the application framework. The JaCoCo Ant tasks and JaCoCo Maven plug-in use the agent and its options directly. The JaCoCo agent collects execution information and dumps it on request or when the JVM exits. There are three different modes of execution data output: File System: At JVM termination execution data is written to a local file. TCP Socket Server: External tools can connect to the JVM and retrieve execution data over the socket connection. Optional execution data reset and execution data dump on VM exit is possible. TCP Socket Client: At startup, the JaCoCo agent connects to a given TCP endpoint. Execution data is written to the socket connection on request. Optional execution data reset and execution data dump on VM exit is possible. [13] Coverage report for the source code Two main coverage metrics which are tracked by JaCoCo are Instructions (C0) and Branches (C1). With JaCoCo are also tracked other coverage metrics like Cyclomatic Complexity, Lines, Methods, and Classes. An example of the JaCoCo report is shown in figure

37 1. Code coverage in the Java ecosystem [13] Figure 1.2: Code Coverage Report for JaCoCo [17] Figure 1.3: Detail of code coverage for a java class from JaCoCo report 21

38 1. Code coverage in the Java ecosystem In JaCoCo report the source code s lines have different colors. A detailed example of a report view is in figure 1.3. These colors have the following meaning: Green lines are fully covered lines of code. Yellow lines are partially covered lines. Red lines are not covered lines of code. These lines were not executed during an analyze run. In JaCoCo report branches in the source code are marked with a diamond with different colors. These diamonds are at the first line of the decision branch. A detailed example of a report view is in figure 1.3. Colors of diamonds have the following meaning: [13] Green diamond indicates fully covered branch. Yellow diamond indicates partially covered branch. That means that branch was executed but was not completed. For example, an exception was thrown out of the decision branch. Red diamond indicates that the decision branch wasn t called during analyze. 22

39 2 Design of detection duplicate code coverage In this chapter is described the solution of detecting duplicate test coverage in Java applications. In the JaCoCo are inserted probes as atomic operations. For the solution is not possible to get the location of the call in atomic time. And it means that solution for detecting of duplicate calls increase the complexity of the whole run. For the solution is necessary to detect duplicates without dependencies on the current run. For the solution is also not possible to use byte array from JaCoCo. If the byte array would be replaced by other data object, with a capability to store duplicates locations into it, then it will increase the consumption of memory and the whole analyze can fail because of OutOfMemoryError exception. It s needed to figure out how to correctly store information about duplicates. It is also needed to store all information as an atomic operation to avoid issues with access to data variables. One of the requirements is not to replace or change the default behavior of JaCoCo and its API. JaCoCo is widely used code coverage tool, and changes in API or JaCoCo s functionality can break compatibility to other projects, which depends on JaCoCo, or to other JaCoCo users. In this chapter we focus on: how to get the location of the call how to filter out the source of the call how to replace bytecode probe in JaCoCo how to track runtime of the calls how to store data how to display a report how to extend JaCoCo with the new future and do not affect other possible performance issues of the design 23

40 2. Design of detection duplicate code coverage 2.1 Obtain a location of the call The first problem is how can be obtained a location of the call, which executes a part of the observed code. JaCoCo works with a compiled application, and it tracks usage of the source call. That means that JaCoCo analyze does not have access to the test of the application and also it can be used different tests to check code coverage (e.g., integration tests). Because we do not know how the structure and content of the tests look like and we need to know how the application arrived at the observed place, the best solution seems to use a stack trace. A stack trace is a Java debugging tool which shows a call stack at a specific time. A stack trace is mainly used with exceptions. It helps to track down causes of the exceptions and solve bugs in code. Stacktrace can be generated manually with a system call so it can show the stack of functions that were called up to that point. The following java code must be called to generate stack trace manually: Thread. currentthread (). getstacktrace (); Stacktrace, after being obtained, is represented as an object array of StackTraceElement objects. Each element represents a single stack frame. All stack frames except for the one at the top of the stack represent a method invocation. The frame at the top of the stack represents the execution point at which was the stack trace generated. [18] StackTraceElement class contains methods getclassname() and getmethodname(). Method named getclassname() returns the fully qualified name of the class. And method named getmethodname() returns the name of the method. Using these two methods, it can be got the name of the class and name of the method, which caused the source code call when the stack trace was generated. The problem is that in stack trace are also elements which shows a necessary system calls for the application run. So it is required to filter out the right calls. From the StackTraceElement is possible to get more information about the call. But methods like getfilename(), which returns the name of the source file containing the execution point, or getlinenumber(), which returns the line number of the source line containing the 24

41 2. Design of detection duplicate code coverage execution point represented by this stack trace element, are not necessary for the further work. 2.2 Filtering out the location of the source In the stack trace is a lot of details about system calls and calls through thy analyzed application. These informations are not needed for the final report. What is needed is only on StackTraceElement, which contains information about the source. Source are tests which are running against the analyzed application. So source is get from the Stack- TraceElement by methods getclassname() and getmethodname(). From these methods can be created source which contains class test name and methods test name (test scenario). What is well know about the location of the source is that in the stack trace it can be found in the bottom part of the stack. It is because on the top of the stack is an element from where was stack generated and at the bottom of the stack is a root of the call. However, the root of the call does not have to be a source location. In the root of the stack is a system call which starts the whole run of analyze (if tests are part of it) or a call which starts test execution against the analyzed application. What should not be forgotten is that test are normal Java classes. That means tests can implement testing interfaces or can have an abstract parent class. So filtering cannot rely on the first source found because it may be an abstract class and this class is not the correct source. It is needed to find the implementation of the abstract class in the stack trace and this will be the correct source location. In the stack, the abstract calls are stored before the implementation call. 2.3 Replacement of the inserted bytecode In actual version of JaCoCo are inserted probes, these probes insert into the code of the application instruction to confirm passage. JaCoCo instrumentation is described on page 13. The current state does not meet the needs to insert other instruction. Because JaCoCo on the level of bytecode it is necessary to add calling of the location as bytecode instructions. By adding a large number of bytecode instructions, there 25

42 2. Design of detection duplicate code coverage is a risk that the code will be not able to run because of issues with stack stability and overflow. Java compiler is designed to compile Java code to the optimal bytecode for application run. To avoid some issues with bytecode instructions added into the code is better to insert into the code calling for external method than extending the actual instructions and complicated logic of the bytecode instruction added by JaCoCo. In the external method can be easily implemented functionality for detecting code coverage duplicates. All the logic of getting the source, filter the source location and saving of the result can be handled by this external class. It may seem like that with added external class is changed call for generating the stack trace, but with external class is added only one element on the top of the stack above the observed place. 2.4 Runtime of the calls Once the tests are completed, the time of each test is known. However, it can not be ascertained how long the single parts of the test lasted. Using this information, it would be possible to detect slow spots, especially bottlenecks, badly merging multi-thread operations, and inappropriate use of wait methods (e.g., Thread sleep). After the in-depth exploration of JaCoCo and comprehension of the functional principles the requirement to track the time for how long the tests were in the specific piece of code can be considered as invalid. It is caused by how JaCoCo works with source code. JaCoCo inserts an atomic operation to the pieces of codes to track their usage. So JaCoCo cannot add timer before the observed part of the code and after it. And JaCoCo does not know the tests, so it is hard to say when is actual test executed and when the test ends. For tracking could be added timestamps when is get a location of the call but this value does not say anything about the runtime of the piece of code, it only says at which time JaCoCo gets the location of the call. Values could be connected with appropriate source locations. But these time values will be affected by getting the location code and filtering out the corresponding test source. 26

43 2.5 Storing of the duplications 2. Design of detection duplicate code coverage In JaCoCo is information about coverage represented by boolean array where probe id is a location of the value in the array. So data about code coverage are represented as primitive value true or false. For extended functionality, when are detected duplicates, is required to store data about all source calls. That means that one place can have from zero to N source calls represented as String, which consists of a full class name and method name. Also storing should be able to handle when is added some source call multiple times because some test can call same code repeatedly (e.g. creating more instances of the tested object) and information about how many times was the piece of code call is not required and storing of this redundant information will be very expensive for memory consumption. Data should be represented by a default Java collection. To preserve the acquired data the most appropriate is a ConcurrentHashMap, which has an implementation of the set configured as a value. Using a probe id, from the map can get the set for storing call locations. To reduce a load on JVM memory and avoid to OutOfTheMemory exceptions data should be saved into external (local) file. To save to a file is necessary to choose a suitable format such as JSON file or local database. Working with non-application storage can increase the overall run time of the analysis. This increase causes access to the file. To improve performance, data can be stored after large parts. 2.6 Report for users with information about duplicate test coverage JaCoCo provides by default well-looking report. For displaying the results of duplicate locations is it needed to extend this report. The JaCoCo report is created from the small parts (Java classes) and then are results merged to the more significant pieces (packages). Smallest parts hold information about code coverage in numerical values which for better understand is represent in percents. Packages then add up and display the average of the all classes and sub-packages. With new storage of data, report do not know which pieces of code were visited (covered). Now from the stored data report know, which 27

44 2. Design of detection duplicate code coverage place was visited from where. This requires changes in generation report for the new functionality of JaCoCo. In the report the missing pieces do not get as false bytes from the array, but now is it represent a part of the code which was not executed by any test class. So in storage data, this case is represented as an empty visitation of the code. 2.7 Extended JaCoCo with the new functionality It is required not to break the actual functionality of JaCoCo. Existing users should not be affected by new changes. They depend on the current version of JaCoCo, and new changes can badly break their configurations. Main code coverage analyzes functionality should stay untouched and works fine after all changes will be done. So the new functionality should not be turned on by default. Therefore it is necessary to implement all new functionality as extensions of the current code, and during the implementation, the JaCoCo API should not be changed. New function added into an application interface is followed by big changes in code, so it is usually required to introduce changes like this with a major release version of the application. All these restrictions help prevent to break backward compatibility. All code added to JaCoCo should have corresponding JUnit test cases. Ideally, tests are developed before or along with the actual implementation. Test cases should verify every new feature. Test cases should also reflect modified behavior. 2.8 Possible performance issues of the design The control flow analysis and probe insertion strategy of JaCoCo allows recording instruction and branch coverage efficiently. In total classes instrumented with JaCoCo increase their size by about 30%. Because probe execution does not require any method calls, only local instructions, the observed execution time overhead for instrumented applications typically is less than 10%. [15] It is expected that performance of JaCoCo with the new functionality will increase. An increase of the size of instrumented classes should not be massive, because inserted probes are similar in proportion to 28

45 2. Design of detection duplicate code coverage the original one. In the JaCoCo extension are used method calls, and local instructions are not used. So observed execution time overhead for instrumented applications by new probes will increase. 29

46

47 3 Implementation In this chapter are introduced new features for expanding the functionality of JaCoCo. Implemented parts are written in a programing language Java. The ASM library is used for bytecode manipulation. One of the requirements is not to replace or change the default behavior of JaCoCo and its API. 3.1 Filtering method First, we must be able to get correct location of the source call. From the probe, we get the array of stack trace elements as an input for the filtering method. We do not need to change the input to any Java collection. An array as the input value is quite suitable for us because mostly in the filtering method we need to access to the stack trace element and get from it a class name and method name. From the array, we will only need to read the values. In arrays, we can access the specific element by its index. Reading stack trace element from the array by its index is a constant operation with a size O(1). And it does not depend on the size of the array. After a detailed examination of the JaCoCo functionality and observation of stack trace generated by the source code of the application during the test, we noticed how the source code was called and thus how it can be filtered out from which test was the code of the application executed. Since we know how JaCoCo was started, it can be used for filtering. Calling the source code of the application that we observed with the probes is due to running tests. Therefore, from the stack trace, it can be observed that the tests are triggered by the reflection from the internal Java API which was triggered by the test runner from the test framework. Reflection is a feature in the Java programming language. It allows an executing Java program to examine or "introspect" upon itself, and manipulate internal properties of the program. For example, it s possible for a Java class to obtain the names of all its members and display them. [19] Runners are used for running test classes. A Runner runs tests and notifies a RunNotifier of significant events as it does so. 31