A Heuristic-based Approach to Identify Concepts in Execution Traces

Transcription

1 A Heuristic-based Approach to Identify Concepts in Execution Traces Fatemeh Asadi * Massimiliano Di Penta ** Giuliano Antoniol * Yann-Gaël Guéhéneuc ** * Ecole Polytechnique de Montréal, Canada ** Dept. Of Engineering Univ. of Sannio, Italy CSMR Madrid (Spain) 1

2 Motivations Software systems lack adequate documentation Developers try to understand systems through Static analyses, visualizations built upon static data Dynamic analyses, requiring the execution of the system (Dynamic) concept identification Identify sets of method calls in execution traces responsible for the implementation of domain concepts or user-observable features Existing approaches based on static analysis [Anquetil and Lethbridge (1998)], dynamic analysis [Wilde and Scully (1995) Tonella and Ceccato (2004)], IR techniques [Poshyvanyk et al. (2007)], or hybrid ones [Eaddy et al. (2008)] CSMR Madrid (Spain) 2

3 Proposed approach A novel approach that analyzes execution traces and groups together method calls that: (i) sequentially invoked together/in sequence (ii) cohesive and decoupled from a conceptual point of view Assumptions Let us consider a feature is being executed in a scenario e.g., Open a Web page from a browser or Save an image in a paint application The set of methods related to the feature is likely to be: (i) conceptually cohesive (ii) decoupled from those of other features (iii) sequentially invoked CSMR Madrid (Spain) 3

4 Proposed approach Step I System instrumentation Step II Execution trace collection Step III Trace pruning and compression Step IV Textual analysis of methods source code Step V Search-based concept identification CSMR Madrid (Spain) 4

5 Step I and Step II Getting Traces Step I - System instrumentation System instrumented using the MoDeC instrumentor MoDeC tool to extract and model sequence diagrams for Java systems Java bytecode instrumentation tool Inserts appropriate and dedicated method invocations in the system to method/constructor entry/exit, points Allows for trace tagging Step II - Execution trace collection We exercise a system following operation sequences taken from user manuals or use case descriptions CSMR Madrid (Spain) 5

6 Step III Trace Pruning and Compression Removing methods not very useful for feature identification Methods occurring in many scenarios Are often utility methods We use the same idea of tf-idf in Information Retrieval Too frequent methods Could be for example related to crosscutting concerns We remove methods having a frequency Q3 + 2 IQR (75% percentile + 2 the interquartile range) Trace compression Aim: collapse repetitions in execution traces Purpose: reduce the search space for Step V Examples: m1(); m1(); m1(); m1(); m2(); m1(); m2(); m1(); m1; m2(); Performed using the Run Length Encoding (RLE) Applied for sub-sequences having an arbitrary length CSMR Madrid (Spain) 6

7 Step IV Conceptual cohesion and coupling determined according to [Marcus et al., 2008] and [Poshyvanyk et al., 2006] Index identifiers, comments contained in methods Extraction of identifiers and comment words Camel-case splitting of composed identifiers Stop word removal (English + Java keywords) Stemming using the Porter stemmer Indexing using tf-idf Reduce the term-document space into a (smaller) conceptdocument space using Latent Semantic Indexing (LSI) Helps to cope with synonymy and homonymy Concept space=50 CSMR Madrid (Spain) 7

8 Step V We use a search-based optimization technique based on Genetic Algorithms (GA) to split traces into segments Representation: a bit-vector where 1 indicates the end of a segment Trace splitting Representation m 1 m 2 m 1 m 3 m 4 m 1 m 4 m 6 m Mutation: randomly flips a bit (i.e., splits or merge segments) Crossover: two-points Selection: Roulette Wheel CSMR Madrid (Spain) 8

9 Step V Quality of the Solution Fitness Function: Segment Cohesion is the average (textual) similarity between any pair of methods in a segment Segment Coupling is the average (textual) similarity between a segment and all other segments in the trace Other GA parameters 200 individuals 2,000 generations for JHotDraw and 3,000 for ArgoUML 5% mutation probability, 70% crossover probability Distributed GA implementation (across 4 servers) CSMR Madrid (Spain) 9

10 Empirical Study Goal: analyze the novel concept location approach based Purpose: of evaluating its capability of identifying meaningful concepts Quality focus: accuracy and completeness of the identified concepts Context: an implementation of our approach and execution traces extracted from two open source systems, JHotDraw and ArgoUML CSMR Madrid (Spain) 10

11 Research Questions RQ1: How stable is the GA, through multiple runs, when identifying concepts into execution traces? RQ2: To what extent the identified concepts match the ones in the oracle? RQ3: How accurate is the identification of concepts in execution traces? CSMR Madrid (Spain) 11

12 RQ1: GA stability We compute the overlap between segmentations obtained in multiple runs using the Jaccard overlap Score Two segments overlaps when they contain calls in the same position of the trace Because a segment of trace T1 overlaps with more segments of T2, the highest similarity is chosen Run 1 Run 2 m 1 m 2 m 1 m 3 m 4 m 1 m 4 m 6 m 1 m 1 m 2 m 1 m 3 m 4 m 1 m 4 m 6 m 1 2/3 2/4 3/4 CSMR Madrid (Spain) 12

13 RQ1: Results Average overlap between 72% and 84% Slightly higher convergence for ArgoUML Ability of the algorithm to converge, despite the relatively large search space CSMR Madrid (Spain) 13

14 RQ2: Matching with the Oracle We manually tag start-end of features while executing the system Using the MoDeC instrumentation tool While executing the instrumented system, the user triggers the introduction of <Start> and <Stop> tags in the trace Matching between identified traces and oracle computed as in RQ1 Run 1 Oracle m 1 m 2 m 1 m 3 m 4 m 1 m 4 m 6 m 1 m 1 m 2 m 1 m 3 m 4 m 1 m 4 m 6 m 1 2/3 2/4 3/4 CSMR Madrid (Spain) 14

15 RQ2: Results High overlap for some features e.g., Draw rectangle or Draw circle Lower for features obtained adapting other ones e.g., Add text obtained adapting Draw rectangle In other cases, low overlap is due to large segments split into more smaller and cohesive ones CSMR Madrid (Spain) 15

16 RQ3: Accuracy in trace identification Computed similarly to RQ2, however we use Precision instead of Jaccard overlap Score Run 1 Oracle m 1 m 2 m 1 m 3 m 4 m 1 m 4 m 6 m 1 m 1 m 2 m 1 m 3 m 4 m 1 m 4 m 6 m 1 2/2 2/3 3/4 CSMR Madrid (Spain) 16

17 RQ3: Results Precision often very high In most cases above 85% and often equal to 100% Low precision (mean 32%) for Add text Relatively low (mean 69%) for Draw rectangle These two features are difficult to be distinguished CSMR Madrid (Spain) 17

18 Inspection of the obtained segments Add class (ArgoUML) The approach split this long feature of 199 methods sequence into 5 segments related to sub-features (creation of objects, adding the project class, handling namespace, setting object properties, handling persistence of the diagram) Create note (ArgoUML) Only the first part (50 methods) of the trace composed of 88 calls was identified Problems related to multi-threading Problems related to collapsing (during compression) loops containing variants Cut rectangle (JHotDraw) Only the last 39 out of 172 calls were included in the segment Methods related to adding to the clipboard and showing the rectangle as cut First methods related to GUI events and split in many small segments Spawn window (JHotDraw) 72 out of 197 methods included The remaining ones were related to setting up menu command properties CSMR Madrid (Spain) 18

19 Threats to Validity Construct validity (relation btw. theory and observation) Multi-threading can change the ordering of calls in multiple executions of the same scenario A better assessment of the actual content of the obtained segments is needed Internal validity (presence of confounding factors) Trace tagging may be imprecise, again due to multi-threading Noise due to utility methods GA intrinsic randomness External validity (generalization of findings) We analyzed two different systems, multiple traces As usual, further empirical evaluation is needed CSMR Madrid (Spain) 19

20 Conclusions We proposed a search-based approach to automatically locate concepts in execution traces By splitting traces into conceptually cohesive and decoupled segments Empirical study on traces from JHotDraw and ArgoUML shows that The approach is stable Identified segments highly precise Finer-splitting wrt. high-level features Limitations due to: multi-threading, GUI events, feature adaptation.. Work-in-progress: Improve performance Use enhanced compression techniques Automatically label identified concepts Perform an extensive empirical validation CSMR Madrid (Spain) 20

21 Thank You! Questions? CSMR Madrid (Spain) 21