24 th Annual Research Review

Transcription

1 24 th Annual Research Review April Towards Better Understanding of Software Quality Evolution Through Commit-Impact Analysis Pooyan Behnamghader USC CSSE

2 Commit-Impact Analysis Software Repository (e.g., Git and SVN) Evolution History Commit Fine-grained incremental changes Contains Information: Code changes - Author, message, time, etc. Each commit a new revision Impact Difference between revisions before and after a commit

3 Impactful Commit Not all changes are interesting! Docs Change Performance Change Main module The module under analysis Usually contains most of the codes core, src/main, framework Excl. tests, docs, plugins, etc. Impactful commit Changes source files in the main module Introduces a new revision of the main module Figure 1. The development history of a simple case study Circles commits Gray circles impactful commits

4 Difference Between Two Revisions Two steps Extracting a model from each revision Defining distance metrics to compare those models Models Programming Analysis Techniques Static PMD, FindBugs, SonarQube absolute values (numbers/vectors) Architecture recovery techniques (ARC, ACDC, etc.) graphs/sets Dynamic Rendering HTML files Distance Metrics Subtracting two numbers/vectors Comparing two graphs/sets

5 Example: Absolute Values

6 Example: Commit-Impact

7 Empirical Study Setup Research Questions: To what extent do developers commit impactful changes? To what extent and how do impactful commits break the compilability of the project? To what extent do impactful commits affect software quality attributes? Should developers rely on a single software quality metric as a change indicator?

8 Data Collection ATLAS Large Scale Evolutionary Analysis on Software Systems System selection process: Fetched all apache projects metadata using GitHub api Name, #commits, repo url, PL, and latest update Only Java Projects At least one update in 2017 Less than 3K commits The main module exists (manual check) No nontrivial prerequisite for compilation (manual check) At least 100 compilable impactful revisions fully analyzed impactful revisions of 38 systems

9 The centrality of the main module 48% of commits are impactful To What Extent do Developers Commit Impactful Changes? 69% of developers commit impactful changes What may significantly impact the ratios of impactful commits and impactful developers? Architecture of the system The level of its integration with other systems Distribution of tasks during the development

10 To What Extent and How do Impactful Commits Break the Compilability of the System? 98% of commits are compilable. Interesting Observations: Relying on snapshot versions of dependencies may cause compilation error. Compiling the project in a new environment from scratch may reveal missing dependencies. After a file is added, compiling the project in a new environment may reveal compilation error. When contributing to the system alone, do compile the system in a new environment once in a while. Committing too early may result in an error commit. Committing too often may introduce error commits. Large refactoring may introduce error commits. Maintenance and code cleanups may introduce error commits.

11 Difference between library systems and others Heavily documented (JavaDocs) Lower LC More stable api Lower FN/LC More stable architecture Lower CS/FN Importance of running tests Complexity (CX) change: 53% Consider maintenance Code smell (SM) change: 46% Importance of running integrated tools To What Extent do Impactful Commits Affect Software Quality Attributes? 64% of subject systems contain PMD plugin Still 60% of impactful commits change PD Only 0.2% commit messages mention using PMD Importance of using multiple techniques to find potential security holes

12 Should Developers Rely on a Single Software Quality Metric as a Change Indicator?

13 Tool Demonstration (Web Application Prototype) JIRA Git

14 Backup Slides

15 Motivation - The main question - How does software evolve? - Version Level Analysis - Versions the major milestones - Too coarse grained - The impact of each developer is omitted - The size of the system drastically increases Other attributes increase