An Extensive Study of Static Regression Test Selection in Modern Software Evolution

Transcription

1 An Extensive Stdy of Static Regression Test Selection in Modern Software Evoltion Owolabi Legnsen 1, Farah Hariri 1, Agst Shi 1, Yafeng L 2, Lingming Zhang 2, and Darko Marinov 1 1 Department of Compter Science, University of Illinois at Urbana-Champaign {legnse2,hariri2,awshi2,marinov}@illinois.ed 2 Department of Compter Science, The University of Texas at Dallas {yxl131230,lingming.zhang}@tdallas.ed ABSTRACT Regression test selection (RTS) aims to redce regression testing time by only re-rnning the tests affected by code changes. Prior research on RTS can be broadly split into dynamic and static techniqes. A recently developed dynamic RTS techniqe called Ekstazi is gaining some adoption in practice, and its evalation shows that selecting tests at a coarser, class-level granlarity provides better reslts than selecting tests at a finer, method-level granlarity. As dynamic RTS is gaining adoption, it is timely to also evalate static RTS techniqes, some of which were proposed over three decades ago bt not extensively evalated on modern software projects. This paper presents the first extensive stdy that evalates the performance benefits of static RTS techniqes and their safety; a techniqe is safe if it selects to rn all tests that may be affected by code changes. We implemented two static RTS techniqes, one class-level and one methodlevel, and compare several variants of these techniqes. We also compare these static RTS techniqes against Ekstazi, a state-of-the-art, class-level, dynamic RTS techniqe. The experimental reslts on 985 revisions of 22 open-sorce projects show that the class-level static RTS techniqe is comparable to Ekstazi, with similar performance benefits, bt at the risk of being nsafe sometimes. In contrast, the method-level static RTS techniqe performs rather poorly. CCS Concepts Software and its engineering Software evoltion; Atomated static analysis; Software defect analysis Software testing and debgging; Keywords regression test selection, static analysis, class firewall Permission to make digital or hard copies of all or part of this work for personal or classroom Permissionse to is make granted digital withot or hard feecopies provided of that all or copies part of arethis notwork madefor distribted personal or for classroom profit orse commercial is grantedadvantage withot fee and provided that copies that copies bear this arenotice made and the or distribted fll citation for profit on theorfirst commercial page. Copyrights advantagefor and components that copiesof bear this this work notice owned and by theothers fll citation than ACM on themst first be page. honored. Copyrights Abstracting for components with credit ofisthis permitted. work owned To copy by otherwise, others thanor ACM repblish, mst be to honored. post on Abstracting servers or to with redistribte credit is permitted. to lists, reqires To copy prior otherwise, specific or permission repblish, to post on servers or to redistribte to lists, reqires prior specific permission and/or a and/or fee. Reqest a fee. Reqest permissions permissions from Permissions@acm.org. from permissions@acm.org. FSE 16, November 13-19, 2016, Seattle, WA, USA FSE 16, November 13 18, 2016, Seattle, WA, USA c 2016 ACM. ISBN /16/11...$ DOI: 1. INTRODUCTION Modern software projects evolve rapidly as developers add new featres, fix bgs, or perform refactorings. To ensre that project evoltion does not break existing fnctionality, developers commonly perform regression testing. However, freqent re-rnning of fll regression test sites can be extremely time consming. Some test sites reqire weeks to rn [32], bt waiting even a few mintes for test reslts can be detrimental to developers workflow. In addition to redcing developers prodctivity, slow regression testing can consme a lot of compting resorces. For example, engineers at Google have observed a qadratic increase in their total test-rnning times [10,15,16], showing that regression testing is challenging, even for a company with a lot of compting resorces. As a reslt, a large body of research has been dedicated to redcing the costs of regression testing, sing approaches sch as regression test selection [13, 20, 27, 28, 31, 41, 45], regression test-site redction [18, 34, 35, 47, 48], regression test-case prioritization [9,19,32,43,44], and test parallelization [6]. Yoo and Harman provide a thorogh srvey of regression testing approaches [42]. Regression test selection (RTS) is the most widely sed approach to speeding p regression testing [10]. RTS aims to redce regression testing efforts by only re-rnning the tests affected by code changes. An RTS techniqe is safe if it selects all tests whose behavior may be affected by code changes; not rnning any of those tests may case developers to miss regressions. Prior research on RTS can be broadly split into dynamic and static techniqes. A typical dynamic RTS techniqe reqires two types of information: (1) changes between two code revisions, and (2) test dependencies dynamically compted while rnning the tests on the old code revision. Given these inpts, the techniqe analyzes how the code changes interact with the dependencies to determine a sbset of tests that may reach (and ths get affected by) the code changes. Dynamic and safe RTS has been drawing attention in the literatre since at least 1993 [30,42], with some newer techniqes sch as DejaVOO [27], FaltTracer [45], and Ekstazi [13]. Different dynamic RTS techniqes differ in precision 1 and analysis overhead. Techniqes that collect finer-granlarity dependencies may be more precise, selecting fewer tests to be 1 Safe techniqes always select all tests affected by code changes bt cold also select some non-affected tests. 583

2 rn, bt can incr higher analysis overhead; in contrast, techniqes that collect coarser-granlarity test dependencies may be less precise bt can have lower analysis overhead. The state-of-the-art dynamic RTS techniqe, Ekstazi [11 13], tracks changes and dependencies at the granlarity level of files; for Java code, these files inclde bytecode classes. Ekstazi comptes (1) classes which changed between revisions and (2) classes that each test class reqired while rnning on the old code revision. Ekstazi selects to rn on the new code revision only tests that depend on at least one of the changed classes. Prior experiments with Ekstazi showed that sing coarse-granlarity test dependencies (at the class level) can sbstantially save the end-to-end testing time (that incldes time to analyze changes, rn selected tests, and pdate dependencies) [13]. De to this, several open-sorce projects (e.g., Apache Camel [1], Apache Commons Math [2], and Apache CXF [3]) have already incorporated Ekstazi into their bild systems [13]. Despite the recent advances in dynamic RTS, its reliance on dependencies collected dynamically cold limit its application in practice, making it important and timely to reconsider static RTS. First, when performing RTS, dynamic test dependencies for the old revision may not always be available, e.g., on the first application of RTS to the project (the project may have earlier revisions). Second, dynamic test dependencies for large projects may be time-consming to collect. Third, for real-time systems, dynamic RTS may not be applicable, becase code instrmentation for obtaining dependencies may case timeots or interrpt normal test rn. Finally, for programs with non-determinism (e.g., de to randomness or concrrency), dependencies collected dynamically may not cover all possible traces, leading to dynamic RTS being nsafe. In contrast to dynamic RTS, which collects test dependencies dynamically, static RTS [5, 8, 22, 33] ses static program analysis to infer, ideally, an over-approximation of the test dependencies to enable safe test selection. However, althogh static RTS techniqes for object-oriented langages have been proposed over three decades ago [8, 22], to or knowledge, these techniqes have not been stdied extensively on modern, real-world projects. In particlar, it is not clear a priori which granlarity level wold be better for static RTS, althogh (i) the most recent work [13] on dynamic RTS shows that class level provides better reslts than method level [45] and (ii) de to the growing and relatively larger scale of modern software systems, other recent state-of-the-art dynamic RTS techniqes [12, 13, 28, 45] se coarser granlarity (e.g., classes and methods) rather than finer granlarity (e.g., statements or CFG edges) [20,30,31], which are more expensive to collect. To investigate the safety, precision, and overhead of static RTS, we implemented one class-level static RTS techniqe and one method-level static RTS techniqe. The class-level static RTS techniqe (ClassSRTS) is or implementation of the previosly proposed class firewall [22] techniqe; it finds class-level dependencies by reasoning abot inheritance and reference relationships in a class dependency graph. ClassS- RTS selects to re-rn any test class that transitively depends on any changed class in the dependency graph. The method-level static RTS techniqe(methsrts) tilizes callgraph analysis [5,17,39]; it constrcts a call graph with all test methods as entry points and selects to re-rn test classes that can transitively reach any changed class throgh a traversal of the call graph. Or ClassSRTS and MethSRTS implementations are based on the ASM bytecode maniplation and analysis framework [4] and the T.J. Watson Libraries for Analysis (WALA) [40], respectively. We evalated these two static RTS techniqes on 985 revisions of 22 open-sorce Java projects. We considered two variants of ClassSRTS and eight variants of MethSRTS, and we compared them against Ekstazi. The reslts show that ClassSRTS has comparable performance as Ekstazi, bt ClassSRTS is occasionally nsafe. In contrast, MethSRTS performs rather poorly: it does not provide performance benefits and is more freqently nsafe. The latter reslt was somewhat srprising as one may expect finer-grain analysis at the method level to be safer and more precise (bt potentially slower) than the coarser-grain analysis at the class level. In conclsion, we recommend that researchers contine improving static RTS techniqes at the coarser granlarity, which already shows promising reslts (at least at the level of classes if not modles or projects). 2. BACKGROUND We introdce the two static RTS techniqes based on program analysis at different granlarity levels: ClassSRTS performs class-level analysis [22] (Section 2.1), and MethSRTS performs method-level analysis [5,17,29,33,39] (Section 2.2). Althogh they perform analyses at different levels, we implemented both techniqes to report selected test classes to aid comparison. In the rest of the paper, when we refer to a test, we mean a test class. Recent srveys on regression testing [42] and change-impact analysis [24,26] provide more details abot static and dynamic RTS. 2.1 Class-Level Static RTS (ClassSRTS) Leng et al. [25] first introdced the notion of firewall to assist testers in focsing on code modles that may be affected by program changes. Kng et al. [22] frther introdced class firewall to accont for the characteristics of object-oriented langages, e.g., inheritance. Given a set of changed classes, a class firewall comptes the set of classes that may be affected by the changes, conceptally bilding a firewall arond the changed classes. The original class firewall techniqe was proposed for the object-relation graph in C++ [22], and Orso et al. [27] generalized it to the intertype relation graph (IRG) to additionally consider interfaces in Java. Sbseqently, we se types to denote classes and interfaces. To the best of or knowledge, sing the IRG and the class firewall is the only proposed techniqe to perform class-level static RTS in Java. An IRG represents the se and inheritance relations between types in a program, as defined by Orso et al. [27]: Definition 2.1 (intertype relation graph). An intertype relation graph, IRG, of a given program is a triple N,E I,E U where: N is the set of nodes representing all types in the program; E I N N is the set of inheritance edges; there exists an edge n 1,n 2 E I if type n 1 inherits from n 2, and a class implementing an interface is in the inheritance relation; E U N N is the set of se edges; there exists an edge n 1,n 2 E U if type n 1 directly references n 2, and aggregations and associations are in the se relations. 584

3 BasedonthisdefinitionofIRG,theclassfirewallisdefined as the types that can (transitively) reach some changed type throgh se or inheritance edges: Definition 2.2 (Class Firewall). The class firewall corresponding to a given set of changed types τ N is compted over the IRG N,E I,E U sing the transitive closre of the dependence relation D = (E I E U) 1 ; firewall(τ) = τ D, where 1 denotes the inverse relation, denotes the reflexive and transitive closre, and is the relational prodct. ClassSRTS takes as inpt the two program revisions and the regression test site T that consists of the tests for the new revision. The otpt is the sbset of tests T s T that may be affected by the changes. ClassSRTS first bilds an IRG, comptes the changed types between two program revisions, and adds the transitive closre of each changed type to the class firewall. Finally, ClassSRTS retrns all tests in the class firewall as the selected test set T s. Note that ClassSRTS need not inclde spertypes of the changed types (bt mst inclde all sbtypes) in the transitive closre becase a test cannot be affected statically by the changes even if the test reaches spertype(s) of the changed types nless the test also reaches a changed type or (one of) its sbtypes. 2.2 Method-Level Static RTS (MethSRTS) A program call graph (CG) represents invocation relationships among program methods [39]. Intitively, starting from each root method (e.g., the main method), the callgraph constrction finds all methods that can be (transitively)invokedfromtherootmethod. Acallgraphisdefined as follows: Definition 2.3 (Call Graph). A call graph CG of a given program is a pair N,E, where: N is the set of all methods in the program nder analysis; E N N are the method invocation edges. MethSRTS takes as inpt the two program revisions and the regression test site. The otpt is T s, the sbset of tests that may be affected by the changes. A test is affected if any of its test methods is affected. (Or experiments show that MethSRTS is rather slow, not becase it selects to rn test classes instead of test methods, bt becase the analysis itself is slow.) Frther, changes are compted at the class level rather than the method level becase recent work [13] demonstrated that class-level changes do not reqire complex modeling of changes de to dynamic dispatch (e.g., sing lookp changes [29, 33]), and can be extremely fast to compte. MethSRTS first bilds a call graph for the old revision sing as root methods all pblic methods (e.g., inclding methods) in the test site. Then, MethSRTS iterates over each test to check if any of its methods may be affected by the changed types τ, i.e., if it can (transitively) reach methods in τ. Finally, MethSRTS retrns the selected set of tests. 2.3 Example Althogh MethSRTS performs analysis at the method level, a finer level of granlarity than ClassSRTS, it does not necessarily deliver more precise RTS. To illstrate both techniqes, consider the example in Figre 1. The class L is in a library, while classes C1 and C2 are in the code nder test. T1, T2, and T3 are three tests. Sppose C2 is modified (marked in gray). A typical dynamic RTS techniqe (e.g., 1 //library code 2 class L { 3 void m1() {} 4 } 5 6 //sorce code 7 class C1 extends L { 8 void m1(){c2.m3()} 9 void m2(){} 10 } 11 class C2 { 12 static void m3(){} 13 } L i C1 C2 1 //test code 2 class T1 { 3 void t1() { 4 L l = new L(); 5 l.m1(); 6 } 7 } 8 class T2 { 9 void t2() { 10 L l = new C1(); 11 l.m1(); 12 } 13 class T3 { 14 void t3() { 15 C1 c = new C1(); 16 c.m2(); 17 } 18 } Figre 1: Example code T1 T3 T2 Class Firewall (a) IRG T1.t1 T2.t2 T3.t3 L.L L.m1 C2.m3 C1.m2 (b) Call graph C1.m1 C1.C1 Figre 2: Example IRG and call graph Ekstazi), will select to re-rn only T2, the only test that exectes the modified C2. We discss the RTS reslts for ClassSRTS and MethSRTS. ClassSRTS: Figre 2(a) shows the IRG for the example. Edges labeled and i are se and inheritance edges, respectively. The class firewall (enclosed in the dashed area) consists of all classes that can potentially reach the modified class, C2. Tests T2 and T3 are in the class firewall and ths selected. Note that T3 is selected de to the imprecision of the class-level analysis: althogh T3 ses C1, it does not invoke any method of C2. MethSRTS: Figre 2(b) shows the call graph for the example. The method C2.m3 in the modified class C2 is marked in gray. From the call graph, tests T1 and T2 can potentially reach C2.m3 and are ths selected. Althogh MethSRTS can be more precise than ClassSRTS in determining that T3 cannot invoke C2.m3, MethSRTS incrs another imprecision even advanced call-graph analyses cannot precisely determine receiver object types in all cases. For example, T1 invokes m1 with the static receiver type L; a naive call-graph analysis (e.g., CHA) will treat all sbclasses of L as potential rntime receiver types. Therefore, MethSRTS may imprecisely select T1 that cold potentially invoke C1.m1 becase C1 extends L. In contrast, ClassSRTS does not have this isse; any possible rntime object type has to be referenced by the test to instantiate it. 2.4 Analysis Scope Modern software projects se external libraries extensively which can slow down static analysis in general and MethSRTS in particlar. For example, in one revision pair of the project invokebinder (p1, Table 5), we fond that sing RTA call-graph analysis takes 370.8% of RetestAll time when third-party libraries are exclded from the analysis, bt it becomes % of RetestAll time when libraries 585

4 are inclded orders of magnitde difference! Therefore, we consider the impacts of exclding third-party libraries for ClassSRTS and MethSRTS as follows. Theorem 2.1. Exclding third-party libraries cannot introdce a new type of safety isse for ClassSRTS. Proof. Exclding third-party libraries removes all types in the library from the IRG. If the exclsion indced a safety isse, there mst be a path from a test T to a changed type C that contains a library type L; otherwise, the exclsion will not impact the selection reslts. This implies a path from T to L and a path from from L to C. However, third-party libraries are bilt before the code nder test, and therefore L cannot statically inherit from or se C 2. Therefore, the path from L to C does not exist. Contradiction. Theorem 2.2. Exclding third-party libraries can introdce a new type of safety isse for MethSRTS. Proof. As shown in Figre 2, by inclding library code MethSRTS can select the trly affected test T2. However, if the library is exclded, Line 11 in Figre 2(b) will be ignored by the call-graph analysis becase the static receiver type is L. Then, T2 only reaches C1.C1() that does not invoke any methods from the code nder test. Ths, T2 does not reach any method in the modified class C2, and wold not be selected. Therefore, exclding third-party libraries can introdce a new type of safety isse for MethSRTS. As a reslt, we exclde libraries for ClassSRTS (making the analysis rn faster while selecting the same set of tests), bt we evalate MethSRTS both with and withot library exclsion to investigate the cost/safety tradeoff. 3. IMPLEMENTATION In this section, we describe the implementation details of or ClassSRTS and MethSRTS techniqes. Details of Ekstazi, the dynamic RTS tool sed in or evalation, can be fond elsewhere [12, 13]. Change comptation: Finding syntactically changed sorce files can be done easily sing the diff tility or version-control systems, bt syntactic changes (e.g., simple reformatting) may not translate to bytecode changes [13]. Therefore, we compte changes at the bytecode level, leveraging the comparison tility from Ekstazi. More specifically, given two program revisions, the comparison first detects the bytecode files that differ between the revisions, and then invokes the Ekstazi API to compte smart checksms [13] (by removing debg-related information) of those bytecode files to frther filter ot the files where only debg-related information changed. Using the Ekstazi change comptation also enables a fairer evalation and comparison of tools. Graph constrction: For ClassSRTS, we sed the ASM bytecode maniplation framework (version 5.0) [4] to constrct the IRG. Or tool ses ASM to parse the bytecode of each (changed) classfile, traversing all the fields, methods, signatres, and annotations to collect all types that are referenced/sed by the type in the classfile. It also collects all 2 Note that even when there are callbacks from the library code, the static reference to the receiver type of the callback is sally referenced by some class from the code nder test to pass to the library code; also note that the safety isses of static RTS cased by reflection (shown in Section 4.3) exist even with library code analysis. types that the type in the classfile extends/implements. Importantly, it incrementally pdates the IRG compted from a prior revision by analyzing only the classfiles that changed. For MethSRTS, we sed the call-graph analyses from the IBM WALA framework [40]. We evalated for widely sed call-graph anlayses: CHA (Class Hierarchy Analysis), RTA (Rapid Type Analysis), 0-CFA(Control-Flow Analysis), and 0-1-CFA, in the ascending order of precision [17, 39]. The analyses effectively differ in how they approximate the rntime types of receiver objects, e.g., CHA does not approximate the rntime types at all, while 0-CFA ses one set of types to approximate the rntime types. In general, a more precise call-graph analysis may incr a higher overhead. Frthermore, WALA also allows exclding the library code from the analysis to speed it p. Therefore, we stdied these for analyses both with and withot library exclsion to investigate the cost/safety trade-offs for MethSRTS. We sed 0-CFA with library exclsion as the defalt MethSRTS variant becase (1) it is recommended by the WALA ttorial [40] for general call-graph analysis applications, and (2) or reslts show it to perform well among all the eight variants for the specific application of call-graph analysis to RTS. Both tools constrct the appropriate graph (IRG or call graph) on the old program revision and serialize the constrcted graph to the disk for the new revision. Graph traversal: Given an appropriate graph (IRG or call graph), and a set of changed nodes, each tool needs to find the tests that can reach the changed nodes. Or tools always traverse the graph representing the old revision. For ClassSRTS, we evalate two modes, offline and online, that traverse the(old) graph at different points. The offline mode comptes transitive closre of the entire graph in advance (before the new revision and the changes are known), and prodces a mapping from test to dependencies; the time to compte the closre is not conted in the end-to-end time. The selection then simply checks what test has some changes among its dependencies. The online mode comptes only nodes transitively reachable from the changes once it knows what those changes are; the time to compte reachability is conted in the end-to-end time. Both modes also incrementally pdate the old graph to prodce a new graph for the next revision; the time to perform the pdate is not conted for offline, while it is conted for online. For Meth- SRTS, all eight variants se the online mode. We did not try the offline mode for MethSRTS becase (1) MethSRTS performs poorly in terms of safety and precision and is not worth frther cost analysis, and (2) MethSRTS selects many more tests than Ekstazi/ClassSRTS so its offline mode can be predicted to be inferior to both others. To implement graph traversals, we se JGraphT [21]. 4. EVALUATION AND RESULTS We present or experimental setp and the reslts of evalating static RTS techniqes in terms of nmber of selected tests, time overhead, precision, and safety. We evalate two variants of ClassSRTS and eight variants of MethSRTS, and we compare them against two baselines: RetestAll (which jst rns all tests) and Ekstazi. Finally, we present some examples of the safety and precision isses of static RTS. 4.1 Experimental Setp To evalate static RTS, we se 22 open-sorce projects, listed in Table 1. We chose these projects among single- 586

5 Table 1: Projects sed in stdy ID Project Name SHA kloc Revs Tests T[s] p1 invokebinder p2 logback-encoder 4fe0f4a p3 compile-testing 8d5229e p4 commons-cli 3ba638a p5 commons-dbtils p6 commons-filepload p7 commons-validator bcb1ec p8 asterisk-java 08dda p9 commons-codec 50a1d p10 commons-compress ec p11 commons p12 commons-collections p13 commons-lang bcb33ec p14 commons-imaging b1fdec p15 commons-dbcp p16 b.hikaricp 19e0c5d p17 commons-io p18 addthis.stream-lib 4dc p19 commons-math 79c p20 OpenTripPlanner aa21c p21 commons-pool p22 jankotek.mapdb eac22b Average Project p1 p2 p3 p4 p5 p6 p7 p8 p9 p10 p11 p12 p13 Ekstazi ClassSRTS MethSRTS modle Maven projects with JUnit 4 tests from (1) the original Ekstazi paper [13], (2) one of or previos stdies [23], and (3) poplar GitHb Java projects with longer-rnning tests (which we deliberately chose becase they are more likely to benefit from RTS). Table 1 shows basic statistics for the projects: SHA is the initial revision of the project on which or experiments started, kloc is the nmber of thosands of lines of code in the project, Revs is the nmber of revision pairs sed in or evalation, Tests is the nmber of tests in the project, and T[s] is the time in seconds to rn all the tests in each project. kloc, Tests, and T[s] are averages across all revisions that we sed. We sed Ekstazi off-the-shelf to compare with static RTS techniqes. To select the revisions, we followed the methodology from the original Ekstazi paper [13]. For each project, we started with the 100 revisions immediately preceding SHA and then chose the sbset of those 100 revisions (1) that compiled, (2) for which mvn test ran sccessflly, and (3) for which Ekstazi ran sccessflly with mvn ekstazi:ekstazi. Starting from the oldest revision in or list, we ran each RTS techniqe on the sccessive pairs of revisions, simlating what an end ser wold have experienced whensinganrtstool. Wemeasrethenmberoftestsselected by each techniqe, the time taken to rn the selected tests, and the time taken by RTS techniqes to perform the selection. The end-to-end time incldes the time to collect dependencies, analyze what tests to rn, and to actally rn the selected tests for the online variants of Ekstazi and the static RTS techniqes. For all the offline variants, the time to collect dependencies is not inclded. We ran all experiments on a 3.40 GHz Intel Xeon E V2 machine with 16GB of RAM, rnning Ubnt Linx LTS and Oracle Java 64-Bit Server version Research Qestions Or stdy aims to answer these research qestions: RQ1: How do static RTS techniqes compare among themselves and with RetestAll and dynamic RTS in terms of the nmber of tests selected to rn? RQ2: How do static RTS techniqes compare among themselves and with RetestAll and dynamic RTS in terms of rntime? RQ3: How do static RTS techniqes compare with a safe p14 p15 p16 p17 p18 p19 p20 p21 p Figre 3: Percentage of tests selected class-level dynamic RTS in terms of precision and safety? RQ4: How do different variants of the MethSRTS inflence the cost/safety trade-offs? RQ1: Tests Selected Figre 3 shows the percentage of tests selected by static and dynamic RTS relative to RetestAll (the black line in the middle of each boxplot is the mean). On average, Ekstazi, ClassSRTS, and MethSRTS select to rn 20.6%, 29.4%, and 43.8% of all tests, respectively. As expected, ClassSRTS and MethSRTS both tend to select a higher percentage of tests than Ekstazi. We frther discss exactly how precise and safe ClassSRTS and MethSRTS are with respect to Ekstazi in Section Srprisingly, the coarsergranlarity ClassSRTS techniqe selects fewer tests than the finer-granlarity MethSRTS techniqe. One reason was discssed in Section 2.3, and more concrete cases are analyzed in Section 4.3. Overall, althogh inferior to the state-of-theart dynamic RTS in terms of the nmber of tests selected, static RTS still selects only a fraction of all tests RQ2: Time Overhead While the nmber of selected tests is an important internal metric in RTS, the time taken for testing is the relevant external metric becase a developer sing RTS perceives it based on this time. We measre time from the point of view of a developer who commits/pshes some code changes and then waits for test reslts before proceeding with other tasks. Specifically, we follow the original Ekstazi experiments [13] 587

6 Table 2: Smmary of end-to-end testing time relative to RetestAll Ekstazi Ekstazi ClassSRTS ClassSRTS MethSRTS (offline) (online) (offline) (online) Project min max avg min max avg min max avg min max avg min max avg p p p p p p p p p p p p p p p p p p p p p p Average Table 3: Safety and precision violations of static RTS compared to Ekstazi Safety Violation % Precision Violation % ClassSRTS MethSRTS ClassSRTS MethSRTS Project revs min max avg revs min max avg revs min max avg revs min max avg p1 0.0 n/a n/a n/a p2 0.0 n/a n/a n/a p3 0.0 n/a n/a n/a n/a n/a n/a p4 0.0 n/a n/a n/a p5 0.0 n/a n/a n/a p6 0.0 n/a n/a n/a 0.0 n/a n/a n/a p7 0.0 n/a n/a n/a 0.0 n/a n/a n/a p8 0.0 n/a n/a n/a 0.0 n/a n/a n/a p9 0.0 n/a n/a n/a p n/a n/a n/a p n/a n/a n/a 0.0 n/a n/a n/a 0.0 n/a n/a n/a p n/a n/a n/a p n/a n/a n/a p n/a n/a n/a p n/a n/a n/a 0.0 n/a n/a n/a p n/a n/a n/a p n/a n/a n/a p n/a n/a n/a 0.0 n/a n/a n/a p p n/a n/a n/a p n/a n/a n/a 0.0 n/a n/a n/a p n/a n/a n/a Average and compare static and dynamic RTS in terms of the end-toend time that incldes time to (1) analyze what tests shold be rn, (2) rn those tests, and (3) collect dependencies for ftre RTS 3. For any RTS techniqe to be beneficial, this end-to-end time mst be less than RetestAll time. Table 2 smmarizes the end-to-end times for static and dynamic RTS relative to RetestAll. Colmns Ekstazi (offline), Ekstazi (online), ClassSRTS (offline), ClassSRTS (online), and MethSRTS represent the time for Ekstazi in offline and online modes, ClassSRTS in offline and online modes, and MethSRTS, respectively. The difference between the offline and online modes of ClassSRTS is explained in Section 3. For Ekstazi, the online mode collects test dependencies dring test rns on the new revision, while the offline mode collects them in a separate phase (hence, the reslts of test rns can be obtained faster, bt the overall machine time sed is higher) [13]. For each techniqe, we show the minimm, maximm, and average time relative to RetestAll. The last row shows the average for each colmn. 3 Note that the dependency collection time is not conted in the end-to-end time of the offline mode. The reslts show that ClassSRTS and Ekstazi both provide benefits over RetestAll (with the average end-toend time across all projects being 62.5% to 68.2% of the RetestAll time). Comparing ClassSRTS and Ekstazi, we find them to be fairly similar in the respective modes. Based on the average time, ClassSRTS slightly otperforms Ekstazi (62.5%to64.0%inthefastest, offlinemodes), btwedonote that Ekstazi is implemented as a plgin for the Maven bild system [12], while or implementation for ClassSRTS is not yet available as a Maven plgin. Therefore, Ekstazi analysis involves some overhead from Maven. (Both ClassSRTS and Ekstazi actally rn tests throgh Maven.) In brief, we find ClassSRTS and Ekstazi to be eqally good based on these experiments. In the ftre, we plan to perform a deeper comparison, especially to determine how mch the ClassS- RTS analysis performed to select tests is faster than the dynamic analysis that Ekstazi ses, becase Ekstazi tends to select fewer tests to rn than ClassSRTS, as seen in Section For both ClassSRTS and Ekstazi, comparing the offline and online modes shows that they do not differ mch, which means that the end-to-end time is, on average, dominated by the time taken to rn the selected tests, rather 588

7 than by the times to either analyze what tests to rn in the case of ClassSRTS or to compte test dependencies in the case of Ekstazi. Finally, we observe that MethSRTS is sbstantially slower, not only mch worse than the other RTS techniqes bt even worse than RetestAll. Overall, we find that static and dynamic RTS at the class-level have comparable performance, while MethSRTS is effectively seless. We also point ot that RTS techniqes provide more benefits for projects with longer-rnning tests. Conting the nmber of projects in which some mode on average performs worse than RetestAll (i.e., the avg is over 100%), we see that ClassSRTS performs worse in two projects, while Ekstazi performs worse in five projects. However, most of those projects have shorter-rnning tests, with all five of these projects having tests that rn in less than 5 sec. For sch projects, one can simply se RetestAll and not attempt any RTS. In fact, we cold have even removed from or evalation sch projects with short-rnning tests, bt we preferred to keep them to highlight that RTS is not appropriate in all cases. An important point is that RTS techniqes shold be not only as fast as possible bt also as safe as possible any RTS techniqe can be simply made faster by not selecting to rn some tests, bt then it risks missing regressions RQ3: Safety and Precision We compare the safety and precision of ClassSRTS and MethSRTS with respect to Ekstazi, becase Ekstazi is a fairly safe and precise dynamic RTS techniqe [13]. Recall that a safe RTS techniqe selects to rn all tests that cold change their behavior de to the code changes between two revisions, and a precise RTS techniqe selects to rn only the tests that cold change their behavior. Table 3 shows a smmary of safety and precision violations, compted as follows. Let E be the set of tests selected by Ekstazi and T be the set of tests selected by another techniqe on some revision. Safety, respectively precision, violations are compted 4 as E \ T / E T, respectively T \ E / E T, to measre how mch a techniqe is less safe, respectively precise, than Ekstazi; lower percentages are better. For each of the for combinations (of two types of violations and two static RTS techniqes), we tablate for metrics: revs is the percentage of all revisions in which the techniqe was nsafe/imprecise (i.e., the percentage was not 0), and min, max, and avg are the minimm, maximm, and average, respectively, percentages of tests missed (for safety) or selected extra (for precision). Intitively, avg captres how bad the safety/precision violations are when they happen in a project. We se n/a when there were no safety/precision violations for the project. The last row shows the average for each colmn; we treat n/a as 0 when compting the overall averages. Table 3 shows several interesting reslts. One srprising reslt for s is that ClassSRTS was rarely nsafe. Only 2 projects had safety violations, averaging 0.2% across all revisions of all projects evalated. (Some example safety violations are discssed in Section 4.3.) Moreover, we fond that MethSRTS is both less safe and less precise than ClassS- RTS, i.e., method-level static RTS is mch less effective than class-level static RTS. On average, across all 22 projects, the 4 We consider the nion of tests selected by both Ekstazi and the techniqe to avoid division by zero in cases where Ekstazi does not select any test bt a static RTS techniqe selects some tests. percentages of revisions in which ClassSRTS incrs safety violations and precision violations are 0.2% and 33.0%, respectively. For MethSRTS, these percentages increase to 10.6% and 55.7%, respectively (10.4 and 22.7 percentage points more nsafe and imprecise, respectively, than ClassSRTS). ClassSRTS is also more effective than MethSRTS in terms of nmber of projects with safety violations: 2 projects for ClassSRTS vs. 14 projects for MethSRTS. (The nmber of projects with a precision violation is the same, 21, for both techniqes.) Comparing the min, max, and avg vales shows the same trend, i.e., when there is a safety or precision violation, MethSRTS is more nsafe or imprecise RQ4: MethSRTS Variants Impacts of call-graph analyses: Table 4 smmarizes the reslts comparing different call-graph analyses for Meth- SRTS. For each analysis, colmns 2 5 present the average percentage of tests selected for all revisions in each project, while colmns 6 9/10 13/14 17 present the safety/precision/overhead information. From the table, we observe three things. First, in general, more precise call-graph analyses tend to select fewer tests. For example, on average, CHA selects 48.8% of tests, while 0-1-CFA selects 43.4% of tests. This is becase some changed nodes may be reachable from tests in imprecise call graphs bt not reachable in precise call graphs. Second, there is no clear trend for safety isses becase the precision of call-graph analyses does not directly impact the sondness of the constrcted call graphs; instead, the safety isses are sally de to reflection and library exclsion. Third, althogh 0-1-CFA has a mch higher time overhead, it has similar precision with 0-CFA while being the most nsafe of all the for variants. 0-1-CFA is very expensive becase it spends a lot of time to approximate possible rntime types of the receiver object for each method invocation [39]. Comparing 0-CFA and 0-1-CFA, 0-CFA has mch fewer safety isses and lower overhead; comparing CHA and RTA, RTA has mch fewer safety isses and lower overhead. Therefore, 0-CFA (also sggested by WALA developers) and RTA seem to be the most sitable for MethSRTS. Impacts of library exclsion: To stdy the impacts of analyzing library code, for each project, we ran WALA with and withot library exclsion on only one randomly selected revision pair. We do not rn for all revisions, becase analysis withot library exclsion is qite slow. We set a timeot of 3 hors for each revision pair. The reslts are shown in Table 5, where DNF means that the rn timed ot. (Note that the corresponding times in tables 5 and 4 do not match becase Table 5 is for only one revision pair for each project.) 0-CFA and 0-1-CFA timed ot for all projects withot library exclsion, so we do not show them. Additionally, the analyses failed de to memory constraints for three projects (p16, p18, and p20); we do not showtheserows. FromtheresltsinTable5, weobservetwo things. First, withot library exclsion, the more expensive and precise RTA analysis does not pay off RTA selects the same nmber of tests as the less expensive CHA in all cases, for projects where neither analyses timed ot. Second, the analysis overhead relative to RetestAll is mch higher withot library exclsion than with library exclsion. For example, for commons-math (p19), CHA overhead is % withot library exclsion bt 527.3% with library exclsion. This happens becase, withot library exclsion, a total of 33,770 classes need to be analyzed for this project, making 589

8 Table 4: Impacts of different call-graph analyses on MethSRTS Tests Selected % Safety Violation % Precision Violation % Time % Project CHA RTA 0CFA 01CFA CHA RTA 0CFA 01CFA CHA RTA 0CFA 01CFA CHA RTA 0CFA 01CFA p p p n/a n/a p p n/a n/a p n/a n/a n/a n/a p n/a n/a n/a n/a p n/a n/a n/a p n/a n/a p p n/a n/a n/a p p p p n/a n/a n/a n/a p n/a n/a p n/a n/a p n/a n/a n/a n/a p n/a n/a p n/a n/a p n/a n/a n/a n/a p Average Table 5: Impacts of library exclsion on MethSRTS (only one revision pair, not all pairs, per project) Tests Selected % Safety Violation % Precision Violation % Time % exclsion no exclsion exclsion no exclsion exclsion no exclsion exclsion no exclsion Project CHA RTA CHA RTA CHA RTA CHA RTA CHA RTA CHA RTA CHA RTA CHA RTA p n/a n/a n/a n/a n/a n/a n/a n/a p n/a n/a n/a n/a n/a n/a p n/a n/a n/a n/a n/a n/a p n/a n/a n/a n/a p n/a n/a n/a n/a p n/a n/a n/a n/a p n/a n/a n/a n/a p n/a n/a n/a n/a p n/a n/a n/a n/a p n/a n/a n/a n/a n/a n/a n/a n/a p n/a n/a n/a n/a p n/a n/a n/a n/a p DNF n/a n/a n/a DNF DNF DNF p n/a n/a n/a n/a p n/a n/a n/a n/a n/a n/a n/a n/a p n/a n/a n/a n/a p DNF n/a n/a n/a DNF DNF DNF p n/a n/a n/a n/a p n/a n/a n/a n/a Average n/a n/a CHA very time consming. The analysis time for RTA is even higher withot library exclsion. In the same project, the analysis for RTA times ot. RTA has sch a high overhead becase it spends significant time to compte the sets for approximating potential rntime types of the receiver object for every method invocation [39]. Overall, these reslts demonstrate that call-graph analyses withot library exclsion are not practical for static RTS de to the high cost and precision isses. In fact, method-level static RTS nfortnately appears impractical in all configrations. 4.3 Qalitative Analysis We discss concrete cases of safety and precision violations where a static RTS techniqe does not select some test(s) that Ekstazi selects or selects some test(s) that Ekstazi does not select. An nsafe RTS techniqe is bad in a non-obvios way becase it can be deceivingly fast bt risk missing regressions; or analysis of safety violations identifies some cases where static RTS techniqes were nsafe dring or experiments. In contrast, an imprecise RTS techniqe is bad in an obvios way becase the imprecision is reflected in the end-to-end time. These examples illstrate crrent limitations of static analysis for RTS and can provide insight into how to improve them in the ftre. class ClassicalRngKttaFieldStepInterpolatorTest extends pblic void interpolationatbonds() { dointerpolationatbonds(decimal64field.getinstance(), 1.0e-15);} abstract class AbstractRngeKttaFieldStepInterpolatorTest: private <T extends RealFieldElement<T>> RngeKttaFieldStepInterpolator<T> setupinterpolator ( ) { RngeKttaFieldStepInterpolator<T> i = createinterpolator(f, t1 > t0, new FEODE<T>(eqn).getMapper()); FieldBtcherArrayProvider<T> p = createbtcherarrayprovider(f, i); } private <T extends RealFieldElement<T>> FieldBtcherArrayProvider<T> createbtcherarrayprovider(field<t> f, RngeKttaFieldStepInterpolator p){ FieldBtcherArrayProvider<T> integrator = nll;... String ipolname = p.getclass().getname(); String integ = ipolname.replaceall("stepinterpolator", "Integrator"); integ = clz.getconstrctor(...).newinstance(f, f.getone());... } protected <T extends RealFieldElement<T>> void dointerpolationatbonds(field<t> f, doble epsilon) { RngeKttaFieldStepInterpolator<T> ipol= setupinterpolator(f, new SinCos<T>(f), 0.0, new doble[] { 0.0, 1.0 }, 0.125); } class ClassicalRngKttaFieldIntegrator extends AbstractFieldInterpolator: i abstract class AbstractFieldIntegrator: i se inheritance Figre 4: ClassSRTS safety violation de to reflection Safety Violations of Static RTS. Safety violations de to reflection: In commonsmath, between revisions and c246b37, ClassSRTS and MethSRTS miss to select nine tests that Ekstazi selects. The relevant change is to the abstract class Ab- 590