PDF. Applying File Structure Inspection to Detecting Malicious PDF Files. Received: November 18, 2013, Accepted: July 11, 2014

Transcription

1 PDF 1,a) 2,b) , MS Rich Text Compound File Binary PDF PDF PDF PDF % PDF Applying File Structure Inspection to Detecting Malicious PDF Files Yuhei Otsubo 1,a) Mamoru Mimura 2,b) Hidehiko Tanaka 2 Received: November 18, 2013, Accepted: July 11, 2014 Abstract: Targeted attacks using document files that contain executable files are popular. We had proposed methods to detect malicious MS document files (Rich Text or Compound File Binary) using file structure inspection. The methods depend on file format, and couldn t detect malicious PDF files. In this paper, focus on features of malicious PDF files that contain executable files to detect them. The features are, for example, the malicious PDF files contain parts that can not be parsed, or the malicious PDF files contain data that is not related to display contents of the PDF files. The experimental result using 164 PDF files that contain executable files shows the effectiveness of the methods. The methods could detect 99.4% of the malicious PDF files in the experiment. The methods are effective over a long time. Because an attacker is almost not able to alter a file structure. Keywords: targeted attack, malware, PDF file, static analysis, detection % % [1] 1 NPA, Chiyoda, Tokyo , Japan 2 IISEC, Yokohama, Kanagawa , Japan a) mjp11001@grips.ac.jp b) dgs104101@iisec.ac.jp MS Rich Text Compound File Binary PDF Portable Document Format MS [2] exploit exe dll c 2014 Information Processing Society of Japan 2281

2 MS PDF PDF PDF PDF PDF PDF PDF PDF 2. PDF PDF exploit PDF exploit PDF 2.1 exploit [3] PDF JavaScript PDF [4] JavaScript HTML JavaScript exploit PDF exploit JavaScript PDF PDF exploit JavaScript Flash PDF exploit PDF 2.2 [5] Handy Scissors [6] PDF PDF PDF 2.3 [7] PDF [8] PDF PDF PDF PDF PDF PDF 3. PDF PDF ISO [9] Adobe Adobe Extensions [10] 3.1 PDF c 2014 Information Processing Society of Japan 2282

3 4 0 obj <</Length 24 /Filter /ASCIIHexDecode>> stream 48656C6C6F2C576F726C6421 endstream endobj 2 Fig. 2 An example of an object. 1 PDF Table 1 Filters that used in malicious PDF files. 1 PDF Fig. 1 A sample PDF file. 4 PDF 1 % 1 %PDF- PDF %%EOF PDF 0 [ ] [ ] obj endobj PDF EOF xref trailer startxref EOF! Z ASCII85 Decode ASCIIHex 2 16 Decode DCT JPEG Decode zlib Flate Flate Decode JBIG2 JBIG2 Decode 3.2 PDF null 5 Stream 3 Stream Stream Stream 2 Stream stream endstream Stream 1 PDF PDF 3 PDF c 2014 Information Processing Society of Japan 2283

4 4. PDF 3 PDF Fig. 3 A sample document structure of a PDF file PDF1.1 PDF /Encrypt Stream PDF PDF1.5 ObjStm Stream Stream Stream PDF ObjStm ObjStm PDF exploit PDF PDF NULL shellcode shellcode PDF PDF PDF 4 PDF 4 c 2014 Information Processing Society of Japan 2284

5 4.2 2 PDF null PDF Stream PDF Stream Stream Stream PDF FlateDecode Stream Stream FlateDecode Stream Stream Stream Stream Stream FlateDecode DCTDecode JBIG2Decode Stream Stream Fig. 4 The algorithm of the test program. Python PDF PDF PDF 1 PDF PDF PDF 40 bitrc4 128 bitrc4 128 bitaes 256 bitaes PDF PDF PDF 2 3 PDF ObjStm 2 ObjStm 2 3 PDF 1 PDF PDF PDF 1 4 % [ ] [ ] obj endobj c 2014 Information Processing Society of Japan 2285

6 xref startxref trailer PDF 2 PDF MS-DOS MS-DOS NULL 256 Byte NT NULL 256 Byte Byte PDF Stream FlateDecode ASCIIHexDecode ASCII85Decode Stream Stream 3 FlateDecode DCTDecode JBIG2Decode Stream Stream PDF PDF PDF 2 2 PDF RAT Table 2 A summary of the speciments. PDF PDF (KB) (KB) , Table 3 Vulnerabilities used by the specimens of APSB / % APSB /17 5.9% APSB / % APSB / % APSB / % APSB /17 5.9% 4 Table 4 An experimental environment. CPU Core i GHz Memory 8.0 GB OS Windows 7 SP1 Memory (VM) 2.0 GB OS (VM) Windows XP SP3 Interpreter (VM) Python pdf APSB11-08 CVE APSB10-21 CVE PDF [11] % 2 contagio clean [12] PDF /164 c 2014 Information Processing Society of Japan 2286

7 5 Table 5 Detection rates of the test program % % % % 6 Table 6 Comparing detection rates with antivirus softwares % T AV % S AV % M AV 5 3.0% T S M AV % 99.4% 0.69 s 5.63 s 6 3.0% 19.5% 3 T S M AV 23.8% 9, PDF 0.2% Public-Key Security Handler PDF PDF PDF [9] AES ObjStm ObjStm 2 3 PDF KByte PDF 10 1 PDF Flate 3 PDF PDF PDF PDF PDF Flate PDF 1 endstream endobj PDF PDF c 2014 Information Processing Society of Japan 2287

8 PDF PDF PDF PDF s 99.4% PDF 0.2% exploit shellcode JavaScript Flash PDF Stream PDF Stream PDF exploit shellcode 1 2 Handy Scissors [5] PDF exploit PDF PDF 1 PDF ISO PDF PDF PDF exploit shellcode exploit shellcode exploit shellcode PDF exploit JavaScript [3], [4] PDF PDF PDF Python Windows Linux 8. PDF PDF 3 PDF exploit shellcode PDF PDF s 99.4% PDF exploit shellcode PDF PDF PDF PDF 5 Stream c 2014 Information Processing Society of Japan 2288

9 PDF PDF PDF PDF [1] meti.go.jp/press/2011/05/ / html [2] MS Vol.55, No.5, pp (2014). [3] Pavel, L. and Nedim, S.: Static Detection of Malicious JavaScript-Bearing PDF Documents Proc. 27th Annual Computer Security Applications Conference, pp (2011). [4] JavaScript Vol.54, No.1, pp (2013). [5] Handy Scissors Vol.54, No.3, pp (2013). [6] RAT Vol.55, No.2, pp (2014). [7] Xu, W., Wang, X., Zhang, Y. and Xie, H.: A Fast and Precise Malicious PDF Filter, Proc. 22nd Virus Bulletin International Conference, pp (2012). [8] Nedim, S. and Pavel, L.: Detection of Malicious PDF Files Based on Hierarchical Document Structure, 20th Annual Network & Distributed System Security Symposium (2013). [9] ISO : Document management - Portable document format - Part 1: PDF1.7, International Organization for Standardization (2008). [10] Adobe: PDF Reference and Adobe Extensions to the PDF Specification (online), available from adobe.com/devnet/pdf/pdf reference.html (accessed ). [11] 2012 Tokyo SOC IBM - Japan (2013). [12] Mila, P.: 16,800 clean and 11,960 malicious files for signature testing and research (online), available from clean-and malicious-files.html (accessed ) Parallel Inference Engine IEEE c 2014 Information Processing Society of Japan 2289