CloPlag. A Study of Effects of Code Obfuscation to Code Similarity Detection Tools. Chaiyong Ragkhitwetsagul, Jens Krinke, Albert Cabré Juan

Transcription

1 CloPlag A Study of Effects of Code Obfuscati to Code Similarity Detecti Tools Chaiyg Ragkhitwetsagul, Jens Krinke, Albert Cabré Juan

2 Cled Code vs Plagiarised Code A result from source code reuse by copying and pasting [maybe with some modificatis] Segments of code which are identical or similar Code maintenance and management In some cases, code cling may violate software license 1 Created in a similar way as code cles but with different intenti Source code plagiarism violates academic regulatis Oracle vs Google law suit 2 [1] A. Mden, S. Okahara, Y. Manabe, and K. Matsumoto, Guilty or Not Guilty: Using Cle Metrics to Determine Open Source Licensing Violatis, IEEE Software, vol. 28,. 2, pp , [2] 2

3 3 What is Obfuscati? Modifying a program while preserving its semantics Can be achieved at 2 levels: Source code Byte code

4 4 Research Questis RQ1: how do current detecti tools perform against code obfuscati? RQ2: what is the best parameter settings and similarity threshold of each tool? RQ3: how do compilati and decompilati facilitate the detecti process? RQ4: can we apply the best parameters and threshold to other datasets effectively?

5 5 Overview of the Empirical Study Java programs are obfuscated at: Source code level Byte code level Combinati of both Several similarity detecti tools are applied to the data set Varying the settings and threshold of each tool Measure performance of each tool

6 6 Tools Obfuscators Decompilers Detectors ARTIFICE ProGuard Procy Kraka Cle SW plagiarism Compressi Others

7 7 Obfuscators ARTIFICE Source code level Renaming, changing loops & cditial statements, changing increment/ decrement statements ProGuard Bytecode level Rename classes, fields, variables to short, meaningless Schulze, S., & Meyer, D. (2013). On the robustness of cle detecti to code obfuscati th Internatial Workshop Software Cles (IWSC)

8 8 Detectors Cle detectors CCFinderX icles Simian, NiCad Deckard Plagiarism detectors JPlag Sherlock, Plaggie Sim ncd-bzlib 7zncd-BZip2 Inclusi Compressi diff, bsdiff py-difflib py-sklearn.cosine_similarity Others * Totally 21 tools ** All tools have to report similarity values (0-100)

9 9 Test Case 1 RQ1: how do current detecti tools perform against code obfuscati? RQ2: what is the best parameter settings and similarity threshold of each tool? A series of small Java programs InfixCverter SqrtAlgorithm Hai Queens MagicSquare

10 10 Test Data Preparati obfuscated source code to be used in detecti phase source obfuscator compiler bytecode obfuscator decompilers obfuscated code ARTIFICE ProGuard Procy inal javac InfixCverter.java SqrtAlgorithm.java Hai.java Queens.java MagicSquare.java Kraka

11 11 Similarity Calculati 5 sets Hai inal Detecti tools 10 files /set obfuscated ccfx* jplag* sim py-difflib similarity report * Most tools have different parameter settings which can strgly affect the results

12 12 Similarity Calculati for Unsupported Tools 0_ 0_.txt 0_.xml Simian GCF File Cverters SimCal _artifice 1_artifice.t xt 1_artifice.x ml Tools using GCF 1 + SimCal include Simian (textual report) icles (RCF format) NiCad (XML report) Deckard (textual report) [1] Wang, T., Harman, M., Jia, Y., & Krinke, J. (2013). Searching for Better Cfiguratis: A Rigorous Approach to Cle Evaluati. FSE 1

13 Similarity Report (ncd-bzlib) 13 Sqrt/ Sqrt/ Squr/ Squr/ InfCv/ InfCv/artifice InfCv/ InfCv/ InfCv/ InfCv/ InfCv/artific InfCv/artifice InfCv/artifice InfCv/artifice Sqrt/ Sqrt/artifice Square/artifice Square/artifice

14 ncd-bzlib with similarity threshold = Sqrt/ Sqrt/ Squr/ Squr/ InfCv/ InfCv/artifice InfCv/ InfCv/ InfCv/ InfCv/ InfCv/artific InfCv/artifice InfCv/artifice InfCv/artifice Sqrt/ Sqrt/artifice Square/artifice Square/artifice

15 ncd-bzlib with similarity threshold = Sqrt/ Sqrt/ Squr/ Squr/ InfCv/ InfCv/artifice InfCv/ InfCv/ InfCv/ InfCv/ InfCv/artific InfCv/artifice InfCv/artifice InfCv/artifice Sqrt/ Sqrt/artifice Square/artifice Square/artifice

16 16 1. Best threshold (T) Find the best threshold (T) of each tool with a specific parameter setting Calculate a sum of false positive and false negative (FP + FN) of all thresholds Choose T with the minimum false results BestThreshold = {T Min(FPT + FNT)}

17 17 Threshold selecti Best threshold = 31 (FP+FN=166) Threshold TP FP TN FN FP+FN Precisi Recall F-measure (F1)

18 ncd-bzlib with similarity threshold = Sqrt/ Sqrt/ Squr/ Squr/ InfCv/ InfCv/artifice InfCv/ InfCv/ InfCv/ InfCv/ InfCv/artific InfCv/artifice InfCv/artifice InfCv/artifice Sqrt/ Sqrt/artifice Square/artifice Square/artifice

19 19 2. Manually-inspected results Pairs that are closest to the threshold are very sensitive to be false positive or false negative We have fixed cost for doing manual inspecti Remove the top 50, 100, and 200 closest to the threshold for manual inspecti Evaluate the new results after removal distance from T = classifier(x,y)-t

20 20 RQ1: how do current detecti tools perform against code obfuscati?

21 21 RQ2: what is the best parameter settings and similarity threshold of each tool? Tools 7zncd-BZip2 Setting s No removal Remove 50 Remove 100 Remove 200 T FP + FN Setting s T FP + FN Setting s T FP + FN Settings m0=2, m0=2, m0=2, m0=2, mx={1,3,5} mx={1,3,5} mx={1,3,5} mx={1,3,5} ncd-bzlib T FP + FN ccfx 1 b=20, t={1..7} 4 90 b=20, t={1..5} 4 b=21, b=20, t={6,7} 3 t={1..7} b=21, b=22,t=7 t={1..7} b=23,t=7 2 b=22, b=24,t={1..7} t={1..7} jplag-java t= t= b=20, t={1..7} b=21, t={1..7} t= t=6 28 t= b=18, t={1..7} b=19, t={1..7} b=20, t={1..5} b=20, t={6,7} b=21, t={1..7} b=22, t={1..7 b=23, t={1..7} t= jplag-text 3 t= t= t= t= simjava 2 r= r= r= r= py-difflib SM_auto junk SM_auto junk SM_auto junk SM_ whitespace _autojunk pysklearn.cosine_similarit y

22 22 Test Case 2 Observati: compiling/decompiling canicalises the data RQ3: how do compilati and decompilati facilitate the detecti process? Experiment Compiled/decompiled versi of the 1st dataset Two different decompilers: Kraka vs Procy Repeat the detecti steps of Test Case 1 and compare the tool performances

23 23 Compiling/Decompiling Process * some tools have different parameter settings which can affect the results 5 sets Hai inal Compile Decompile Procy Detecti tools similarity report 10 files /set obfuscated javac Kraka ccfx* jplag* sim py-difflib similarity report

24 24 RQ3: how do compilati and decompilati facilitate the detecti process?

25 25 Original ARTIFICE (source-code obfuscated versi)

26 26 Original (decompiled) ARTIFICE (decompiled)

27 27 Test Case 3 RQ4: can we apply the best parameters and threshold to other datasets effectively? Experiment Munich dataset ctaining simis independently developed Java programs for address validati [1] Juergens, E., Deissenboeck, F., & Hummel, B. (2011). Code similarities beyd copy & paste. Proceedings of the European Cference Software Maintenance and Reengineering, CSMR, 78 87

28 28 RQ4: can we apply the best parameters and threshold to other datasets effectively? Tools Settings Test Case 1 (2,500) Test Case 3 (munich) (11,881) T FP+FN T FP+FN ccfx b=20, t={1..7} ,797 simjava r= ,680 jplag-text t= ,770 py-difflib SM_auto junk ,446 7zdcd-Bzip2 m0=2, mx={1,3,5} ,432 ncd-bzlib ,754 jplag-java t= ,162 py-sklearn.cosine_similarity ,282

29 29 Summary Current tools behave differently obfuscated code Cle and plagiarism detectors outperform the others The best parameter settings and threshold can be found using the proposed method Compiling/decompiling can help canicalise the obfuscated code The derived settings and threshold cant be applied directly to other data sets

30 30 What s next? Replicate the experiment other data sets and compare the best parameter settings and thresholds SOCO (detecti of SOurce COde re-use) The Java collecti ctains 259 source codes The C collecti ctains 79 source codes Find a better way to learn the best parameter settings and threshold in ad-hoc manner for each data set (or pair)

31 Questis? 31