Software Clone Detection. Kevin Tang Mar. 29, 2012

Transcription

1 Software Clone Detection Kevin Tang Mar. 29, 2012

2 Software Clone Detection Introduction Reasons for Code Duplication Drawbacks of Code Duplication Clone Definitions in the Literature Detection Techniques and Tools Applications and Related Research

3 Introduction Code cloning reusing code fragments by copying and pasting Studies show that about 5% to 20% of a software systems can contain duplicated code What is difficult? Detect clone with modifications Differentiate original from copies

4 Reasons for Code Duplication (1) Development Strategy Reusing existing code by copying and pasting (with or without minor modification) Forking reuse of similar solutions with the hope that they will be diverged significantly Merging of two similar systems

5 Reasons for Code Duplication (2) Maintenance Benefits Risk in developing new code Ensuring robustness in life-critical systems High cost of function calls in real time programs

6 Reasons for Code Duplication (3) Overcoming Underlying Limitations Writing reusable code is error-prone Difficulty in understanding large system Lack of ownership of the code to be reused Wrong method of measuring developer's productivity

7 Reasons for Code Duplication (4) Cloning By Accident Coincidentally implementing the same logic by different developers

8 Drawbacks of Code Duplication Increased probability of bug propagation Increased probability of introducing a new bug Increased probability of bad design Increased difficulty in system maintenance and improvement

9 Clone Definitions in the Literature Code Clone Types Textual Similarity: Based on the textual similarity we distinguish the following types of clones Type I Type II Type III Functional Similarity: If the functionalities of the two code fragments are identical or similar Type IV

10 Clone Definitions in the Literature Code Clone Types Type I: Identical code fragments except for variations in whitespace (may be also variations in layout) and comments if (a >= b) { c = d + b; // Comment1 d = d + 1; }else{ c = d - a; //Comment2 } if (a>=b) { // Comment1' c=d+b; d=d+1; }else{ // Comment2' c=d-a; }

11 Clone Definitions in the Literature Code Clone Types Type II: Structurally/syntactically identical fragments except for variations in identifiers, literals, types, layout and comments if (a >= b) { c = d + b; // Comment1 d = d + 1; }else{ c = d - a; //Comment2 } if (m >= n) { // Comment1' y = x + n; x = x + 5; //Comment3 }else{ y = x - m; //Comment2 }

12 Clone Definitions in the Literature Code Clone Types Type III: Copied fragments with further modifications. Statements can be changed, added or removed in addition to variations in identifiers, literals, types, layout and comments if (a >= b) { c = d + b; // Comment1 d = d + 1; }else{ c = d - a; //Comment2 } if (a >= b) { c = d + b; //Comment1 e = 1; //new statement d = d + 1; }else{ c = d - a; //Comment2 }

13 Clone Definitions in the Literature Code Clone Types Type IV: Two or more code fragments that perform the same computation but implemented through different syntactic variants int i, j=1; for (i=1; i<=n; i++){ j=j*i; } int factorial(int n){ if (n == 0){ return 1; }else{ return n * factorial(n-1); }

14 Detection Techniques and Tools Text-based Techniques (1) The target source program is considered as sequence of lines/strings Code fragments are compared with each other to find sequences of same text/strings Little or no transformation/normalization is performed on the source code before starting the comparison; in most cases, the raw source code is used directly

15 Detection Techniques and Tools Text-based Techniques (2) Problems Line Break Identifier changes Parenthesis removal/adding for a single statement Fragment 1: 1: dwframegrouplength = 1; 2: for (dwcnt=2; dwcnt<=64; dwcnt*=2) 3: { 4: if ((( uloutrate/dwcnt) * dwcnt)!= 5: uloutrate) 6: { 7: dwframegrouplength *=2; 8: } 9: } Fragment 2: 1: framegrouplength = 1; 2: for (Cnt = 2; Cnt <= 64; Cnt *=2){ 3: if (( ( Rate / Cnt) * Cnt)!= rate) 4: framegrouplength *=2; 5: }

16 Detection Techniques and Tools Text-based Techniques (3) Improvements the following filtering and/or transformation / normalizations are applied Comments Removal: Ignores all kinds of comments in the source code depending on the language of interest Whitespace Removal: Removes tabs, and new line(s) and other blanks spaces Normalization: Some basic normalization can be applied on the source code

17 Detection Techniques and Tools Text-based Techniques (4) Normalization operations on source code elements Operation Language element Example Replacement 1 Literal string abort 2 Literal character y. 3 Literal integer Literal decimal Identifier counter p 6 Basic numerical type int, short, long, double Example: num 7 Function name Main() foo() int increment(int counter) => num foo(num p)

18 Detection Techniques and Tools Token-based Techniques (1) The entire source system is parsed / transformed to a sequence of tokens Scan for finding duplicated subsequences of tokens CCFinder Each line of source code is divided into tokens by a lexer then concatenated into a single token sequence Each identifier related to types, variables, and constants is replaced with a special token A suffix-tree based sub-string matching algorithm is then used to find the similar sub-sequences A mapping is required for obtaining the clone pair information with respect to the original source code

19 Detection Techniques and Tools Token-based Techniques (2) Other improvements Suppress insignificant token classes (e.g., access modifiers of Java) that may cause noise in detection (RTF) Assign the same ID to different types int, short, long, float, double (RTF) Frequent subsequence mining technique where a frequent subsequence can be interleaved in its supporting sequences (CP-Miner) Token-based techniques are also used in the area of plagiarism detection

20 Detection Techniques and Tools Tree-based Techniques (1) Source code is pared to a parse tree or an abstract syntax tree (AST) Search for similar subtrees with some tree matching techniques Return corresponding source code of the similar subtrees found CloneDR Compares subtrees by characterization metrics based on a hash function through tree matching

21 Detection Techniques and Tools Tree-based Techniques (2) Evans and Fraser proposed structural abstraction Suppose the clone a[?] = x; occurs twice: a[i] = x; // lexical a[i+1] = x; // structural

22 Detection Techniques and Tools AST + Suffix Tree Techniques The AST nodes are serialized in preorder traversal A suffix tree is created for these serialized AST nodes Cut the AST node sequences according to their syntactic region Find clones on transformed AST using sequence matching algorithm

23 Detection Techniques and Tools PDG-based Techniques Program Dependency Graph (PDG) contains the control flow and data flow information of a program Isomorphic subgraph matching algorithm applied for finding similar subgraphs PDG-DUP finds isomorphic PDG subgraphs using program slicing group identified clones together while preserving the semantics of the original code for automatic procedure extraction to support software refactoring PDG-based approaches are robust to reordered statements, insertion and deletion of code, but they are not scalable to large size programs

24 Detection Techniques and Tools Comparison of the detection approaches Approach Portability Precision Recall Scalability Textbased High, needs lexer at most 100%, No false positives as checks for exact copies Low, only finds exact copies Depends on Comparison algorithms Tokenbased Medium, Needs lexer transformation rules Low, due to normalization and/or transformation returns many false positives High, can detect most clones High with suffixtree algorithm Treebased Low, needs parser High, parse-tree considers structural info also Low, cannot detect all types of clones Depends, how comparison is made PDGbased Low, needs PDG generator High, considers structural and semantic info too Medium, cannot detect all clones Low, graph matching is costly

25 Detection Techniques and Tools Clone Detection Tools Tool Supported Language Approach Background Dup C, C++, Java Line based/text based Jplag C, C++, Java Token/Greedy String Academic Academic CloneDr C, C++, JAVA, COBOL AST/Tree matching Commercial DupLoc Language Independent Line/Exact string matching Academic PDG-DUP C, C++ PDG/Slicing Academic

26 Applications and Related Research for Clone Detection Plagiarism detection Origin analysis and software evolution Multi-version program analysis Bug detection Malicious software detection

27 Question Does the else-block contain clones of ifblock? If it does, what is the type of the clone? What detection technique can be applied to detect these clones? if (a > 10) i = i + 1; d = a * sin(b); M(i) = d; else if (a > 5) j = j+2; %increment d = b * sin(b); M(i)=d; else j = j + 1; d = a * sin(b + 1); M(i) = d; end k = k + 1; end

28 Question Given a MATLAB source code file containing exactly 1 outer if-statement. Write a program to print all Type II clones of if-block in else-block. You may assume no empty statement / line. Hint: ifblock = ifstmtnode.getifblock(0).getchild(1); elseblock = ifstmtnode.getelseblock().getchild(0); if (a > 10) i = i + 1; d = a * sin(b); M(i) = d; else if (a > 5) j = j+2; %increment d = b * sin(b); M(i)=d; else j = j + 1; d = a * sin(b + 1); M(i) = d; end k = k + 1; end

29 References Chanchal Kumar Roy, James R. Cordy. A Survey on Software Clone Detection Research pdf Magdalena Balazinska, Ettore Merlo, Michel Dagenais, Bruno Lagüe, Kostas Kontogiannis. Advanced Clone-Analysis to Support Object-Oriented System Refactoring