Identifying the Applied Obfuscation Method towards De-obfuscation

Transcription

1 Identifying the Applied Obfuscation Method towards De-obfuscation Hayato Sagisaka Division of Frontier Informatics, Graduate School of Kyoto Sangyo University. Haruaki Tamada Faculty of Computer Science and Engineering, Kyoto Sangyo University. Abstract Recently, to prevent cracking, the various protection methods have been proposed. One of the protection methods is the obfuscation method. Obfuscation method changes the program into hard to understand for hiding secret information in the program. On the other hand, de-obfuscation is an interesting research topic for protecting the software. Since, though vulnerable protection methods are dangerous, measuring the robustness of the method was not discussed. In this paper, we tackle to identify the applied obfuscation methods towards de-obfuscation. To perform de-obfuscation requires a suitable method for each obfuscation method. For this, we obfuscated programs by two practical tools and three algorithms from one academic tool. Then, we analyzed the programs and extracted from them based on opcodes. By using the proposed method, we could identify the applied obfuscation method. I. INTRODUCTION Serious security incidents have arisen, such as illegal analysis and software theft. To prevent such malicious attacks, many protection methods were proposed. However, the robustness of the protection methods were never discussed. Increasing the robustness needs a more deeper discussion about robustnesses and vulnerabilities. Especially, systematizing how to attack to the protection methods helps evaluation and construction of more robust protection method. This paper focuses on de-obfuscation which converts an obfuscated program to more understandable program. If strong de-obfuscation method is established, fragile obfuscation methods will be gone. We consider the process to apply the de-obfuscation method. To get the de-obfuscated program, we apply the reverse transformation of the obfuscation algorithm. The each obfuscation method have different algorithm. That is, reverse transformation also proper for each obfuscation method. Therefore, in the first step, we must identify the applied obfuscation method in order to de-obfuscate. This paper proposes the identifying method for the applied obfuscation method towards de-obfuscation. We develop three research questions to evaluate the proposed method. RQ What does the obfuscation method have? RQ2- Are the useful to identify the known obfuscation method applied to an unknown product? RQ2-2 Are the useful to identify the unknown obfuscation method applied to known product? We conducted the empirical studies to answer the above research questions. The result of the studies shows that the proposed method could identify the applied obfuscation method based on the survey of the obfuscation methods in advance. The remainder of this paper is organized as following. Section II reviewed related works. Section III describes the proposed method and illustrates its process. Section IV presents the empirical studies to answer the research questions. Section V shows conclusion and some future works. II. RELATED WORK The previous protection methods are categorized into () preventing the theft techniques, (2) detecting stolen programs, and (3) proving the theft[]. In category (), there were proposed obfuscation methods, and anti-assemble methods[2], [3]. Also, category (2) includes software birthmark methods[4], [5], and category (3) includes software watermark methods[6]. An attack method against obfuscation methods were proposed for category ()[7]. Cimato et al. proposed the conversion method to de-obfuscate the names in a program. Blanc et al. also proposed the de-obfuscating the program to clarify its control flow with abstract syntax tree[8]. Both methods are quite important, since those methods become one of technique to evaluate the robustness and vulnerabilities of the obfuscation methods. However, to perform de-obfuscation requires a suitable method for each obfuscation method. Futhermore, the applied protection methods are generally private information. Therefore, identifying applied protection methods are important to the first step of the deobfuscation. III. PROPOSED METHOD A. Key Idea Various obfuscation methods are proposed in a few decade. Almost methods convert the given program in order to hide the private information in the program. Each coversion characterize the obfuscation algorithm to hide a certain information. For example, name obfuscation hide means of variables, methods, and class names by changing meaningless names[2]. Because, we assume that the names in the program greatly help to understand the program. Threfore, the name obfuscation method protects the program by converting to more /6/$3.00 copyright 206 IEEE ICIS 206, June 26-29, 206, Okayama, Japan

2 incomprehensible program. Some algorithms are proposed in the name obfuscation methods how to change the names. Straightforward algorithm is to change variable names into one alphabet characters. More powerful algorithm changes variable names into reserved words, and invisible characters such as spaces, tabs, control sequences, and etc[9]. Other algorithms change method names to introduce overloading[2], and change not definition parts of symbol names, but use parts with dynamic name resolution mechanism[0], []. Even though the obfuscation algorithms categorized into the name obfuscation methods, we will extract of each algorithm from the conversion process and resultant programs. Similarly, the case of using other protection methods, of the method will arise. Therefore, we tackle to identify the applied obfuscation method towards deobfuscation by measuring the program artificiality. B. Overview of the Proposed Method The birthmark methods have been proposed, which use of the program[4], [5]. Birthmark method originally computes similarities between the programs with the extracted native in order to detect pirated programs. The process of the proposed method is ported from birthmark methods, which is to extract the of the program, and to compare them. However, extracting are absolutely different. The birthmark methods focus on the of the program. The proposed method focuses on the of the protection method. On the other hands, Kanzaki et al. proposed the method to measure the stealthiness of the program[2]. The method extracts n-gram from the opcodes of the given program, and computes artificiality by the probability from the corpus. The artificiality means how natural opcode sequences are in the target program. Kanzaki et al. assume that a compiler generates natural opcode sequences. Generally, protection methods alter the opcode sequence by their algorithm. Then, the protected program may have unnatural opcode sequences. If a protection p q method f method f 2 method f method f 2 p =f 2 (p) p 2 =f 2 (p) q =f (q) q 2 =f 2 (q) C(p ) Evaluation aspect of the proposed method C(p 2 ) C(q ) Resilient property (Birthmark) C(q 2 ) Fig. : Key idea of the proposed method Credibility property (Birthmark) C(p) C (q) method has natural opcode sequences, the algorithm of the method is stealthy. We focus on the method to compute artificiality, and assume that protected program have a certain of the protection algorithm. Then, we tackle to identify the protection method of the program by the extracted. Figure shows the key idea of the proposed method. In the figure, we obtain obfuscatd program p and p 2 from original program p by certain obfuscation algorithm f and f 2 (p = f (p),p 2 = f 2 (p)). Then, using extraction function C, we extract certain C(p), C(p ), and C(p 2 ) from p, p, and p 2. The evaluation aspects of the birthmark method are resilient property and credibility property. To evaluate resilient property compares C(p) and C(p ), then the high score shows that p and p are in the copy relation. Also, credibility property compares C(p) and C(q), then the low score shows that p and q are developed independently. The evaluation aspect of the proposed method is C(p ) and C(p 2 ), and C(p ) and C(q ). A low score between former relation (C(p ) and C(p 2 )) shows different protection methods were applied. Also, high score between latter relation (C(p ) and C(q )) shows the same protection method was applied. C. Characteristics of the Proposed Method To implement the proposed method, almost birthmark methods would fail to identify the protection method. Because, if we apply a certain protection method f to different programs p and q, birthmark method expects C(p) =C(q). However, the proposed method expects C(p ) C(q ). We focus on the artificiality evaluation method by Kanzaki et al. to the proposed method. A certain protection method inserts ridiculous opcode sequence into never executed block. Another certain protection method changes the natural opcode sequence to unnatural one. Such unnatual opcode sequence might be the of the protection method. Originally, Kanzaki et al. computed the artificiality by probability. Ohtaki et al. ported Kanzaki s method for an object oriented language, such as Java[3]. The method uses perplexity metric instead of Kanzaki s probability for computing the artificiality. The perplexity metric shows the complexity of the natural language. Low perplexity means that the next word is easy to expect. On the other hand, a certain protection method duplicates original natural opcode sequence. Such duplicated sequence is still natural. Therefore, we also focus on the frequencies of the n-gram. Then, we employ the perplexity metric and frequency of the n-gram as the of the protection method. IV. EMPIRICAL STUDIES A. Overview In this study, we conduct three experiments to answer the three research questions. RQ What does the obfuscation method have?

3 (a) Original ASM 000 (b) ALL (Allatori Java Obfuscator) 000 (c) PG (ProGuard) (d) DR (Duplication Registers) (e) IRR (Irreducibility) (f) MLI (Merge local integers) Fig. 2: Histogram of ASM opcdoe in 5-gram (Sorted by original 5-gram frequencies) TABLE I: Target jar files Product Version Product Version ASM 3.3. FakeHack.0 Jhstop 0.0. Jwhich.0 JCalendar.3.3 Robocode-setup.6.0. TABLE II: Obfuscation tools Tools (Algorithms) Abbr. Description Allatori Java Obfuscator ALL Commercial tool. ProGurad 2 PG Popular OSS obfuscation tool. Sandmark 3 Academic protection tool which have many protection algorithms. Duplication registers DR duplicate register for variables. Merge local integers MLI merge two integer values into one long value. Irreducibility IRR update the control flow to irreducible one. RQ2- Are the useful to identify the known obfuscation method applied to an unknown product? RQ2-2 Are the useful to identify the unknown obfuscation method applied to known product? In the first experiment, we extract n-grams of practical protection tools, and investigate the. The second experiment tried to identify the known protection algorithm. The third experiment is similar with second experiment, tried to identify the unknown protection algorithm. To conduct the experiments, we chose the products shown in Table I. Also, the adopted obfuscation tools are shown in Table II. The first column represents tool names and algorithm names of Sandmark. The column named Abbr. shows abbreviation of the tools and algorithms. The last column shows short descriptions of each tool and algorithm. We conducted the experiments on the Java platform, because many obfuscation algorithms are implemented. Extracted are n-gram of opcodes, and computes their perplexities and frequencies. The n value is 2, 3, 4, and 5. At first, we constructed the corpus to compute perplexity. Therefore, we downloaded 3,786 jar files which are distributed by the apache software foundation 4. Note that, the total number of classes is 660,465, and the total number of methods is 4,272,567. B. Analysis of n-gram In this experiment, we try to characterize the protection methods. At first, we obfuscate 6 products shown in Table I by five obfuscators shown in Table II. Next, we extract n-grams of opcode sequence from gotten 30 products and 6 original products. Finally, we investigate the of each tool. The histogram of opcode frequencies is shown in the figure 2. The horizontal axis represents each 5-gram, and the vertical axis shows their frequencies in logarithm. The order of the horizontal axis of all graphs is the same as original histogram. The histogram of original and PG look the same. From investigation of their 5-grams, both 5-grams were just the same. We also investigated 5-grams from other products, the result was almost the same as 5-gram from original. The main algorithm of PG does not affect opcode sequences. Therefore,

4 TABLE III: ASM opcodes in 5-gram before/after obfuscation (top 5 frequencies) Original ALL DR MLI IRR Rank 5-gram Frequency Perplexity <code> <code> <code> <code> aload <code> <code> <code> aload getfield aload getfield ifnull aload getfield istore iload aload arraylength if_icmpge aload ldc invokespecial goto aload aload dup getfield swap getfield aload getfield ldc invokestatic invokevirtual ldc invokestatic aload invokevirtual ifeq goto ldc invokestatic aload invokevirtual dup istore iload if_icmpge aload iconst_0 dup istore istore iload dup istore istore iload aload istore istore iload aload arraylength invokevirtual pop iconst_0 dup istore pop iconst_0 dup istore istore dup2x2 lxor lconst_ lneg bipush lload dup2x2 lxor lconst_ lneg aload lload bipush lshr l2i bipush lushr land lxor lstore lconst_ lneg bipush lushr land i2l iconst_0 i2l lcmp iconst_ i2l lcmp iconst_ ixor iconst_ iconst_0 i2l lcmp iconst_ ixor iconst_ ixor iconst_ iand nop iconst_2 irem istore iload i2l TABLE IV: Characteristics of the protection methods Tools ALL PG DR MLI IRR Characteristics for identifying Swap two original opcodes by swap instruction. Almost the same as original opcode sequence. The proposed method cannot identify this tool. Duplicate istore instruction. Emerge rare 2-gram dup2x2 lxor. Emerge nop instruction. the proposed method are not useful for such obfuscation methods, which keep opcode sequences. On the other hand, histogram of other tools is dramatically different from original histogram. The results show that the protection methods may affect the opcode sequences by their. Table III shows 5-gram of ASM (top 5 frequencies). On the Table III, we omitted PG, since just the same as original 5- grams. From the table III, frequencies were decreased except MLI, and perplexities became a quite high value. The result shows that obfuscated opcode sequences have rare opcode fragments, and their fragments would be. We found of the protection tools through investigation of every product, shown in the Table IV. Unfortunately, the of PG were not found, therefore, other identifying method is required. Table IV is the answer of RQ. C. Identifying the known protection method In the previous experiment, we found the of each tool. In this experiment, we try to identify the known protection method to the unknown products. The known protection method means that the were already extracted (a) Histogram of 5-grams from original product (b) Histogram of 5-grams from obfuscated product (c) Histogram of junit opcdoe in 5-gram (Sorted by original 5-gram frequencies) We chose JUnit as the unknown product, and known protection methods are randomly selected from Table II except PG. Then, we obfuscated junit-3.7.jar and obtain junit-3.7-obfuscated.jar. Next, we extract 5-grams from junit-3.7-obfuscated.jar and computes frequencies and perplexities of 5-grams. Figure 3a shows histograms of 5-grams extracted from 5

5 TABLE V: 5-gram of obfuscated JUnit (top 5 frequencies) 5-grams PPL dup2x2 lxor lconst_ lneg bipush lload dup2x2 lxor lconst_ lneg bipush lushr land lxor lstore lconst_ leng bipush lushr land lneg bipush lushr land lxor Freq. TABLE VI: 5-grams of obfuscated ASM 5-grams PPL iconst_2 irem iconst_0 if_icmpne aload dup dup dup imul imul dup dup imul imul isub dup imul imul isub iconst_ iload dup dup dup imul junit-3.7.jar and junit-3.7-obfuscated.jar. At this time, the obfuscation algorithm was unknown, however, we suppose that some obfuscation algorithms were applied from the graphs. Then, we compare to the graph of figure 3b and graphs in figure 2. The figure 3b seems to be similar with DR (Figure 2d) and MLI (Figure 2f) from the shapes of the graphs. However, we did not succeed identifying the obfuscation method yet, since the graph shapes are not matched exactly. Then, we inspect the details of opcode sequences based on of protection methods shown in Table IV. Before inspection, we filtered target 5-grams as follows: Let P = {g,g 2,..., g l } be a n-gram set extracted from original opcode sequence. Also, let P o = {g o,g o 2,..., g o m} be a n-gram set from obfuscated opcode sequence. Then, we filter the 5- grams in the set difference P o \ P = {x P o x P }. The resultant 5-grams of top 5 frequences are shown in Table V. The column named Freq. means frequencies of corresponding 5-gram. Also, the column named PPL shows perplexities of corresponding 5-gram. From Table V, high perplexity 5-grams are appearing such as including dup2x2 lxor. We inspected resultant 5-grams by the shown in Table IV. Then, we infer that the MLI was applied. Finally, the proposed method succeeded to identify the obfuscation method, since the applied obfuscation method was MLI. Therefore, the answer of RQ2- is yes. D. Identifying the unknown protection method In this experiment, we try to identify the unknown protection method. The unknown protection method means that we never extract the from it. At first, we chose ASM 3.3. as the target products, and SOP (simple opaque predicate) as the unknown protection method. SOP is a protection algorithm implemented in Sandmark. SOP algorithm inserts the opaque predicate (the if statement always true or false) into the programs. At first, we obfuscated asm-3.3..jar and obtained asm-3.3.-obfuscated.jar. Next, we extracted n- Freq. grams from both jar files in the same way as Section IV-C. The histogram of 5-grams is shown in Figure 4. The graph of figure 4 is similar with graph of figure 2d. However, the graphs are not matched. Therefore, we inspect the 5-grams from asm-3.3.-obfuscated.jar. The resultant 5-grams of top 5 friequencies are shown in the Table VI, and we inspect them with the shown in Table IV. From Table VI, all of 5-grams have high perplexities, and include multiple dup, and imul instructions. The of the protection method is not appeared in Table VI. Therefore, we cannot identify the applied obfuscation method. However, we can infer some protection method was applied since the resultant 5-grams did not appear in original opcode sequences. Naturally, we were able to identify the SOP by extracting beforehand in the same way as Section IV-B. Therefore, the answer of RQ2-2 is no. However, extracting from many protection methods strengthens the proposed method. V. CONCLUSION This paper proposed the method to identify the applied protection method by artificiality evaluation method towards deobfuscation. We developed three research questions to evaluate the proposed method. In the empirical studies, we examined the proposed method to answer the three research questions. The answer of RQ is shown in the Table IV, the answer of RQ2- is yes, and the answer of RQ2-2 is no. However, the our method becomes useful gradually by extracting of many protection methods beforehand. In this paper, the of ProGuard could not extract. To support such protection tools is left to our future work. It is also interesting research topic to identify the obfuscation method automatically. REFERENCES [] C. Collberg and J. Nagra, Surreptitious Software: Obfuscation, Watermarking, and Tamperproofing for Software. Addison-Wesley, July [2] P. M. Tyma, Method for renaming identifiers of a computer program, United States Patent 6,02,966, August 2000, filed: Mar.20, 998. [3] A. Monden, Y. Takada, and K. Torii, Methods for scrambling programs containing loops, IEICE Transactions on Information and Systems, vol. 80, no. 7, pp , July 997, (In Japanese). [4] H. Tamada, M. Nakamura, A. Monden, and K. Matsumoto, Design and evaluation of birthmarks for detecting theft of java programs, in Proc. IASTED International Conference on Software Engineering (IASTED SE 2004), February 2004, pp , innsbruck, Austria Fig. 4: 5-gram of obfuscated ASM with unknown algorithm

6 [5] H. Tamada, M. Nakamura, A. Monden, and K. Matsumoto, Java birthmarks detecting the software theft, IEICE Transactions on Information and Systems, vol. E88-D, no. 9, pp , September [6] C. Collberg and C. Thomborson, Software watermarking: Models and dynamic embeddings, in Proc. Principles of Programming Languages 999, POPL 99, January 999, pp , (San Antonio, TX). [7] S. Cimato, A. D. Santis, and U. F. Petrillo, Overcoming the obfuscation of java programs by identifier renaming, Journal of Systems and Software, vol. 78, pp , October [8] M. D. Preda, M. Madou, K. D. Bosschere, and R. Giacobazzi, Opaque predicates detection by abstract interpretation, in In Proc. the th International Conference on Algebraic Methodology and Software Technology (AMAST2006), July 2006, pp [9] PreEmptive Solutions, Java obfuscation DashO, http: // (Last Access: 205/06/22). [0] H. Tamada, M. Nakamura, A. Monden, and K. Matsumoto, Obfuscating API library names using dynamic name resolution, IEICE Transactions on Information and Systems, vol. J90-D, no. 0, pp , October 2007, (In Japanese). [] H. Tamada, M. Nakamura, A. Monden, and K. Matsumoto, Introducing dynamic name resolution mechanism for obfuscating system-defined names in programs, in Proc. IASTED International Conference on Software Engineering (IASTED SE 2008), February 2008, pp , (Innsbruck, Austria). [2] Y. Kanzaki, T. Onoue, and A. Monden, A method for evaluating the stealthiness of protected code based on its artificiality, IPSJ Journal, vol. 55, no. 2, pp , February 204, (In Japanese). [3] R. Ohtaki, T. Ohdo, H. Tamada, Y. Kanzaki, and A. Monden, Stealthiness evaluation of obfuscation methods with opcode and operand of java bytecode instructions, in Proc. of Symposium on Cryptography and Information Security 204 (SCIS204), January 204, pp. CD ROM (2D2 2), (In Japanese).