Software Quality Understanding by Analysis of Abundant Data (SQUAAD)

Transcription

1 Software Quality Understanding by Analysis of Abundant Data (SQUAAD) By Pooyan Behnamghader Advisor: Barry Boehm ARR 2018 March 13,

2 Outline Motivation Software Quality Evolution Challenges SQUAAD Interactive Interface ARR 2018 March 13,

3 Motivation We are interested in analyzing impact of Each change, or a series of changes/tasks/events. A developer over a period of time. To have a better understanding of Software quality evolution. Conflicts and synergies among software quality attributes. To help Organizations determine which divisions and project types have better or worse quality; which quality attributes are being achieved poorly or well; and how do these correlate with customer satisfaction and total cost of ownership. Managers better understand which types of projects or personnel contribute most to quality problems or excellence, and which types of project events correlate with which types of quality increase or decrease. Developers continuously monitor software quality and improve software maintainability. ARR 2018 March 13,

4 Focus Of The Analysis Do all changes matter? Change in the documentations while analyzing performance? Change in the sample applications while analyzing code quality of the core module of a framework? Change in tests while analyzing software architecture? First, we need to determine what part we want to analyze! The module containing domain logic? Tests? Only.java source files? We pick a module to analyze! /. containing the whole repository. /google-oauth-client/src/main/java containing the domain logic. This helps better understanding of heterogeneous projects, developed In different programming languages Apache Avro is implemented in more than 10 programing languages in the same repository. By different development teams. Apache Parquet-MR contains different Java sub-projects; and each sub-project has its own set of developers and reviewers. ARR 2018 March 13,

5 Impactful Developers Definition A developer is impactful if she changes the main module at least once. Org. Time Span Sys. Netflix 05/11-01/18 12 Apache 01/02-02/18 38 Google 08/08-01/18 18 Total 01/02-02/18 68 ARR 2018 March 13,

6 Impactful Commits Definition A commit is impactful if it changes the source code in the main module. Org. Time Span Sys. Netflix 05/11-01/18 12 Apache 01/02-02/18 38 Google 08/08-01/18 18 Total 01/02-02/18 68 ARR 2018 March 13,

7 Compilability Compilability is a basic indication of quality. Syntax errors are simply symptoms of careless development! Some static analysis techniques depend on the availability of byte-code. e.g., FindBugs Running software to conduct dynamic analysis requires producing the byte code. e.g., Test coverage A recent study declares the unavailability compiled versions as the main unresolved source for the manual effort in software evolution analysis. ARR 2018 March 13,

8 Compilability (cont.) Disabled tests/checkers. Targeted the main module. Inspected each uncompilable revision and fixed missing dependencies. 652 completely uncompilable revisions detected! Org. Sys. Dev. Cmts MSLOC Netflix Apache Google Total ARR 2018 March 13,

9 Compilability Tool Assistance Broken Next Broken Next Broken Parent Next Breaker Next Length Period Dev. 12/29/05 7:28 12/29/05 23:55 A frame Fix thi b b8bd3f1 c4fd3de dev_2 dev_ /4/05 12:38 5/4/05 12:57 Fixed b Add res 1e027c5 4ffb78b a17464e dev1 dev /4/05 6:53 5/4/05 12:38 Whitesp Fixed b 4ffb78b 9707c44 1e027c5 dev_2 dev1 5/4/05 3:10 5/4/05 6:53 Add uti Whitesp 9707c44 e21843f 4ffb78b dev_2 dev_2 5/3/05 15:32 5/4/05 3:10 Better Add uti e21843f 6a9871b 9707c44 dev1 dev_2 5/3/05 14:07 5/3/05 15:32 Update Better 6a9871b 45b5cf7 e21843f dev1 dev1 5/3/05 14:05 5/3/05 14:07 Rewrite Update 45b5cf7 aa1b8c5 6a9871b dev1 dev1 ARR 2018 March 13,

10 Uncompilable Sequence of Commits Length: Number of commits in the sequence. Period: How long sequence lasts (min, hour, day, ). Developers: How many developers are involved. ARR 2018 March 13,

11 Developer s Affiliation Does the type of developer s affiliation with an organization affects its contributions? (ongoing research...) Org. Diversity Affiliation Compilability Google 23.76% Google Others P-Value Apache 73.53% Apache Others P-Value Netflix 86.00% Netflix Others P-Value ARR 2018 March 13,

12 Software Quality Metrics ARR 2018 March 13,

13 The Whole Picture! ARR 2018 March 13,

14 Probability Chart The probability for a metric to change while another does not! ARR 2018 March 13,

15 Developer s Affiliation And Quality Metrics This is subject to further investigations (ongoing research) (but preliminary results are interesting!) Org. Aff. LC CX SM VL FG Google Google Others P- Value Apache Apache Others P-Value Netflix Netflix Others P-Value ARR 2018 March 13,

16 Challenges Subject systems Unsuitable case studies e.g., the software is too small to be meaningful for some specific analysis. Different project structure for each system/version e.g., the core module located in a different subdirectory in older versions. Analysis Implementing tool for a new technique e.g., your algorithm is wrong or there is a bug in the implementation.you realize it after observing weird evolutionary patterns in your data. Using off-the-shelf implementation of a solid technique The implementation is simply buggy (e.g., non-deterministic behavior). Off-the-shelf tool s server crashes after analyzing hundreds revisions. Different analysis techniques have different set-up requirements. report format (htmls, json, xml, etc). Comparing results Needs a common meta-model and a query system. ARR 2018 March 13,

17 Challenges (cont.) Commit history Older revisions depend on old build technologies. Revisions use snapshot versions of their dependencies between releases. Some libraries are just missing and require manual downloading. Weird dependencies (e.g., a file inside the bare git folder or the local machine of a developer) Can be computationally expensive. Visualization Hundreds of developers and thousands of changes. Teamwork Someone runs analysis using different configurations Someone targets the wrong modules/subsystems (e.g., tests are also included) Being Unlucky! Power outage in the middle of running analysis. ARR 2018 March 13,

18 I Had a Dream... For each case study, we could declare... The remote repository. The build command(s). The subject modules, subsystems, or packages. The version sets. For each analysis, we could declare how to... Prepare the environment for the analysis. Run analysis on each system/version. Interpret and compare the results, and generate statistics. We could define a portable workflow to automatically run the study... On a powerful remote server. Over a distributed infrastructure (cloud). We could visualize the data... ARR 2018 March 13,

19 SQUAAD An automated cloud-based infrastructure to Retrieve a subject system s information from various sources (e.g., commit history and issue repository). Distribute hundreds of relevant revisions on multiple cloud instances, efficiently compile each revision, and run static/dynamic programming analysis techniques on it. Collect and interpret the artifacts generated by programming analysis techniques to extract quality attributes or calculate change. A set of statistical analysis techniques tailored for understanding software quality evolution. Simple statistics, such as frequency of code smell introduction or correlation between two quality attributes. Machine learning techniques, such as clustering developers based on their impact. An extensible web interface to illustrate software evolution. ARR 2018 March 13,

20 Architecture Source Code Source Code Source Code Compile Compile Configuration Compile Configuration Configuration Brute Brute Force Force Compiler History Compiler Cloud Instance Cloud Instance Recovery Unit Source Code Source Code + Binary Source Files Code + Binary Files + Binary Files ASAT ASAT Analysis Wrapper Artifacts Components NoSQL Datastore Analysis Analysis Result Analysis Result Result Dataflow Connector Analysis Server Analysis Result Analysis Analysis Result Result Orchestrator Report Comparator Report Parser Value Vectors Change Metric Values Web Server Interactive Web Interface Repository Host (GitHub) Git Repository Project/User Metadata Git Commit Analyzer GitHub Analyzer Project/Users Information Commit Information Relational Database Data Master Data Analyzer Issue Repository (Jira) Project Issue Repository Jira Analyzer Issue Information Plotter ARR 2018 March 13,

21 Related Frameworks Program Analysis Techniques Year Type Compilation Distributed SQUUAD 18 Static/Dynamic Yes Cloud 16/17 Static - HPC HistoryMiner [35] 17 Static - - Candoia [37] 17 Static - - Lisa [1] 17 Static - - QualBoa [12] 17 Static - - Boa [14,41] 15/17 Static - - MetricMiner [35] 13 Static - - ARR 2018 March 13,

22 Properties of Studies Empowered by SQUAAD Replicability The ability to define a solid workflow for the analysis. Data consistency and reusability The ability to use the same subject in/for different analyses. Extensibility The ability to run complex static/dynamic analysis in a cloud instance. Scalability The ability to run large-scale analyses on multiple versions. ARR 2018 March 13,

23 External Validity of MSR Studies Heavy re-use of data sets We are able to grow our data set from one study to another. Non-availability of data sets Our dataset is available as NoSQL and Relational Databases. Non-availability of implementations All tools we use are open-source. The framework itself is solid but is not available to public yet. Small data sets Our data sets are comparable to similar state-of-the-art studies. Diverse tooling All integrated tools follow a similar workflow. ARR 2018 March 13,

24 Evolution Trend of a Metric How a single quality attribute evolves. Two metrics Size (top) Code Smells (bottom) One project A period of 9 years ARR 2018 March 13,

25 Impact of Developers How developers impact software quality. Two metrics Size (top) Code Smells (bottom) One project A period of 9 years ARR 2018 March 13,

26 Other Visualizations Coevolution How multiple quality attributes coevolve. Two metrics Duplicate Code Blocks Security Vulnerabilities Normalized values One project A period of 3 years Evolution graph around a data point One metric Security Vulnerabilities One project A period of 1 week ARR 2018 March 13, Node represents commits Edges represent parent-child relationships between commits Values on Edges represent the impact of each commit Red commits are the ones that introduce new issues.

27 SQUAAD Web (v1) ARR 2018 March 13,

28 SQUAAD Web (vision) ARR 2018 March 13,

29 SQUAAD Web (v2) ARR 2018 March 13,

30 Application of The Web Interface Advanced Software Safety Engineering Course Description USC partnered two USC Schools to give the US Navy at China Lake, CA, an Advanced Software Safety Engineering Course with Security and Anti-Counterfeiting highlights in September 2017, with the next being in Usage Analyzed two open-source software complexes. Depicted the evolution of more than 50 metrics in the web interface. Impact Least amount of efforts for the managers and front-end acquisition engineers. Data collection Data interpretation In-depth analysis different quality attributes. ARR 2018 March 13,