Model-based Analysis of Source Code Repositories

Transcription

1 odel-based Analysis of Source Code Repositories arkus Scheidgen artin Schmidt Joachim Fischer Humboldt Universität zu Berlin 1

2 Agenda Software Evolution, Reverse Engineering, and ining Software Repositories odel-based ining of Software Repositories srcrepo a framework for model-bases analysis of software repositories Experiments with Eclipse s software repositories Conclusions 2

3 software engineering Software Evolution software maintenance software modernization transformation requirements user {C} S problem user {C} R HEAD S reverse engineering {C} S S {C} forward engineering mining software repositories 1 R 0 quality assessment implicit dependencies OG s AD (Achitecture-Driven odernization) AST eta-odel (AST) Knowledge Discovery eta- odel (KD) Software etrics eta-odel (S) 1. H. Kagdi,.L. Collard, J.I. aletic: A survey and taxonomy of approaches for mining software repositories in the context of software evolution; Journal of Software aintenance and Evolution: Research and Practice; Vol.19/Nr.2/2007 3

4 ining Software Repositories In General The term mining software repositories (SR) has been coined to describe a broad class of investigations into the examination of software repositories. The premise of SR is that empirical and systematic investigations of repositories will shed new light on the process of software evolution. [1] Different scopes, e.g. single software projects vs. many software projects Different goals, e.g. quality assessments and implicit dependencies vs. generalizations about software evolution 1. H. Kagdi,.L. Collard, J.I. aletic: A survey and taxonomy of approaches for mining software repositories in the context of software evolution; Journal of Software aintenance and Evolution: Research and Practice; Vol.19/Nr.2/2007 4

5 odel-based ining of Software Repositories S {C} reverse engineering 1. E.J. Chikofsky, J.H. Cross: Reverse engineering and design recovery: A taxonomy; IEEE Software; Vol.7/Nr.1/ R. Lincke, J. Lundberg, W. Löwe: Comparing Software etrics Tools; 8th International Symp. on Software Testing and Analysis;

6 odel-based ining of Software Repositories S {C} reverse engineering R HEAD {C n } R 0 1. E.J. Chikofsky, J.H. Cross: Reverse engineering and design recovery: A taxonomy; IEEE Software; Vol.7/Nr.1/ R. Lincke, J. Lundberg, W. Löwe: Comparing Software etrics Tools; 8th International Symp. on Software Testing and Analysis;

7 odel-based ining of Software Repositories S {C} reverse engineering R HEAD {C n } {C R 0 n-1 } 1. E.J. Chikofsky, J.H. Cross: Reverse engineering and design recovery: A taxonomy; IEEE Software; Vol.7/Nr.1/ R. Lincke, J. Lundberg, W. Löwe: Comparing Software etrics Tools; 8th International Symp. on Software Testing and Analysis;

8 odel-based ining of Software Repositories S {C} reverse engineering R HEAD {C n } {C R 0 n-1 } {C 0 } 1. E.J. Chikofsky, J.H. Cross: Reverse engineering and design recovery: A taxonomy; IEEE Software; Vol.7/Nr.1/ R. Lincke, J. Lundberg, W. Löwe: Comparing Software etrics Tools; 8th International Symp. on Software Testing and Analysis;

9 odel-based ining of Software Repositories S {C} reverse engineering R HEAD {C n } {C R 0 n-1 } {C 0 } 1. E.J. Chikofsky, J.H. Cross: Reverse engineering and design recovery: A taxonomy; IEEE Software; Vol.7/Nr.1/ R. Lincke, J. Lundberg, W. Löwe: Comparing Software etrics Tools; 8th International Symp. on Software Testing and Analysis;

10 odel-based ining of Software Repositories Scope depends on concreter SR-application and its goals number of software projects: single repositories, large repositories, ultra-large repositories Sources as text and text based metrics, e.g. LOC Declarations only: packages, classes, methods, but no statements, expressions, etc. Full AST with or without cross-references 6

11 odel-based ining of Software Repositories Scope depends on concreter SR-application and its goals number of software projects: single repositories, large repositories, ultra-large repositories Sources as text and text based metrics, e.g. LOC Declarations only: packages, classes, methods, but no statements, expressions, etc. Full AST with or without cross-references 7

12 odel-based ining of Software Repositories SR tools are already model-based, but in a proprietary manner Idea: existing reverse engineering framework and corresponding standard meta-models and modeling frameworks instead of proprietary solutions Goals deal with heterogeneity (different version control systems, different languages) reuse of existing meta-models, transformations, and languages interoperability with existing analysis tools retaining meaningful scalability 8

13 odel-based SR Strategies version control system - -B 3 9

14 odel-based SR Strategies Checkout(r) version control system - -B 3 9

15 odel-based SR Strategies Checkout(r) version control system - -B 3 9

16 odel-based SR Strategies B 3 Checkout(r) version control system - -B 3 9

17 odel-based SR Strategies B 3 d2cus(r) Checkout(r) version control system - -B 3 9

18 odel-based SR Strategies B 3 d2cus(r) Checkout(r) version control system - -B 3 9

19 odel-based SR Strategies B 3 d2cus(r) Checkout(r) version control system - -B 3 9

20 odel-based SR Strategies B 3 d2cus(r) Analysis(r) 1 Checkout(r) version control system - -B 3 9

21 odel-based SR Strategies B 3 d2cus(r) Analysis(r) 2 1 Checkout(r) version control system - -B 3 9

22 odel-based SR Strategies B 3 3 d2cus(r) Analysis(r) 2 1 Checkout(r) version control system - -B 3 9

23 odel-based SR Strategies B 3 3 d2cus(r) Analysis(r) 2 1 Checkout(r) version control system - -B 3 R Checkout + CUs Parse + Analysis!! 9

24 odel-based SR Strategies B 3 3 d2cus(r) Analysis(r) 2 1 Checkout(r) version control system - -B 3 10

25 odel-based SR Strategies B 3 3 d2cus(r) Analysis(r) 2 1 Checkout(r) d2 CUs(r) f B.f -B 3 version control system - B.f f 10

26 odel-based SR Strategies B 3 3 d2cus(r) Analysis(r) 2 1 Checkout(r) d2 CUs(r) B.f f erge(r) -B 3 version control system - B.f f 10

27 odel-based SR Strategies B 3 3 d2cus(r) Analysis(r) 2 1 Checkout(r) d2 CUs(r) B.f f erge(r) -B 3 version control system - R B.f f Checkout +! Parse + Analysis CUs Checkout +! + erge)+analysis CUs(Parse R 10!

28 odel-based SR Strategies B 3 3 d2cus(r) Analysis(r) 2 1 Checkout(r) d2 CUs(r) B.f f erge(r) -B 3 version control system - B.f f 11

29 odel-based SR Strategies B 3 3 d2cus(r) Analysis(r) 2 1 Checkout(r) d2 CUs(r) B.f f erge(r) -B 3 version control system - Save(r) B.f f Load(r) 11

30 R R R B 3 CUs odel-based SR Strategies Checkout + CUs Parse + Analysis Checkout + CUs(Parse + erge)+analysis (Load + erge)+analysis 0 d2cus(r)!!! Analysis(r) Checkout(r) d2 CUs(r) B.f f erge(r) -B 3 version control system - Save(r) B.f f Load(r) 11

31 R R R B 3 CUs odel-based SR Strategies Checkout + CUs Parse + Analysis Checkout + CUs(Parse + erge)+analysis (Load + erge)+analysis 0 d2cus(r)!!! Analysis(r) Checkout(r) d2 CUs(r) B.f f erge(r) version control system - -B 3 Save(r) B.f f Load(r) Analysis 0 (r) 12

32 R R R R B 3 B CUs 1 odel-based SR Strategies Checkout + CUs Parse + Analysis Checkout + CUs(Parse + erge)+analysis! (Load + erge)+analysis CUs (Load + Analysis 0 ) d2cus(r)!! Analysis(r) Checkout(r) d2 CUs(r) B.f f erge(r) version control system - -B 3 Save(r) B.f f Load(r) Analysis 0 (r) 12

33 odel-based SR Strategies B 3 3 d2cus(r) Analysis(r) 2 1 Checkout(r) d2 CUs(r) B.f f erge(r) version control system - -B 3 Save(r) B.f f Load(r) Analysis 0 (r) 13

34 Research Questions Assumptions Development of SR-applications based on models, transformation languages and standardized meta-models is favorable Some SR-applications need to analyze source code on a deep (AST) level SR-analysis is performed iteratively Hypotheses odels of source code repositories can be created and persisted Traversing existing persistent models of source code repositories is much faster than traversing transient models that are created from version control system on the fly 14

35 srcrepo A Framework for odel-based SR Eclipse s odisco as reverse engineering framework reverse engineering for Java, based on EF Support for many JRE-ased languages: Java, xtext, JSP, L creates instances of a Java EF meta-model that corresponds to the handwritten JDT AST-model provides transformation to language independent artifacts, e.g. KD EF-Fragments 1 to store very large-models uses No-SQL databases and stores larger model fragments within database entries in contrast to object-by-object stores such as OR-based CDO or No-SQL-based orsa or Neo4J tend programming with higher order functions to mimic OCL-style definition of software metrics 2 1..Scheidgen, A.Zubow,J.Fischer,T.H.Kolbe: Automated and Transparent odel Fragmentation for Persisting Large odels; AC/IEEE 15th International Conference on odel Driven Engineering Languages & Systems (ODELS); Innsbruck; Scheidgen, J.Fischer: odel-based ining of Software Repositories; 8th Systems and odeling Conference, Valencia, Spain, September 29th,

36 OCL to Calculate etrics of AST-odels // Weighted number of methods per class. def wmc(abstracttypedeclaration type,(block) int weight) { } type.bodydeclarations.sum[if (body!= null) weight.apply(it.body) else 1] 16

37 Experiments Eclipse Foundation sources, i.e. Eclipse platform and plugins (large scale software repository) Organized in different (couple hundred) projects: jdt, cdt, emf,... Available via GITHub GIT repositories can be gathered automated via GITHub s REST-ful API 200 largest Eclipse repositories that actually contained Java code: 6.6 GB Git, 400 LOC, 250 GB model with 4 billion objects. 17

38 Example Plot: Halstead-length for each Revision; Eclipse CDT WC with Halstead length (x 10 6 ) time (years)

39 odel Create v Analysis Times time (hours) udf merge/increment load save parse checkout avg time per revision (ms) jdt.ui xtext eclipselink jdt.core swt cdt ocl ptp org.aspectj cdo checkout parse save load merge/ increment udf 19

40 Diskspace GIT repository vs model size 15 GIT size odel size 10 jdt.core cdt jdt.ui cdo in GB 5 0 f.emfstore.core 20 emf jdt.debug emf.compare emf.texo.diffmerge.core

41 Delta-Compression 1 Named element matching eta-class matching Line matching delta-models initial revisions uncompressed detla-models initial revisions uncompressed delta-lines initial revisions uncompressed GB GB Lines cdt webtools gmf tooling rap jdt.core cdt webtools gmf tooling rap jdt.core cdt webtools gmf tooling rap jdt.core Compressed relative to full size Avg. execution times Avg. execution times (logarithmic) (%) avg. time per revision (ms) avg. time per revision (ms) initial revisions delta models for named elements delta models for meta-class delta lines parse compress named elements compress decompr. meta-class named elements decompr. meta-class parse compress named elements compress decompr. meta-class named elements decompr. meta-class 1..Scheidgen: Evaluation of odel Comparison for Delta-Compression in odel Persistence; BigDE 2016 (at STAF 2016),Vienna, Austria, July 6-7,

42 odel Creation v Analysis with Delta-Compression without compression with compression udf merge/increment load save parse checkout time (hours) udf merge/increment load save compress parse checkout jdt.ui xtext eclipselink jdt.core swt cdt ocl ptp jdt.ui xtext eclipselink jdt.core swt cdt ocl ptp org.aspectj cdo org.aspectj cdo time (hours)

43 Conclusions SR can support software evolution and helps to understand software evolution Traversing a source code repository to gather information (SR) is very time consuming, especially with iterative analysis It is possible to save most of this time via saving data in its model state, at the cost of comparably large models that need to be persisted The SR analysis execution time savings are considerable 23