CodeXt: Automatic Extraction of Obfuscated Attack Code from Memory Dump

Size: px

Start display at page:

Download "CodeXt: Automatic Extraction of Obfuscated Attack Code from Memory Dump"

Rosemary Pierce
5 years ago
Views:

1 CodeXt: Automatic Extraction of Obfuscated Attack Code from Memory Dump Ryan Farley and Xinyuan Wang George Mason University Information Security, the 17 th International Conference ISC 2014, Hong Kong Where Innovation Is Tradition Oct , 2014

2 Problem Need to automate upon detection in memory Avoid substantial manual effort Automatically recover malcode Extract/unpack/recover attack code Memory dump, transient artifacts Input Memory Dump Process Context Registers Execution Trace Log Files Obfuscated Code Analysis Engine Static Dynamic Hybrid Output Attack String Malicious Code Vulnerability Arbitration Obfuscation Removal Normalized Code 1

3 Existing Tools Only work with known boundaries Typically designed for full binaries e.g., PE files Things get nasty without given boundaries Or are arbitrary byte streams Don t generically handle Malformed, Misaligned Obfuscated, Armored 2

4 Solution: CodeXt Discovers executable code within memory dump Upon realtime detection Extracts packed or obfuscated malcode First to generically handle Incremental and Shikata-ga-nai Decoder3 w/ K3 Decoder3 w/ K3 Decoder3 w/ K3 Decoder3 w/ K3 Encoded code, data Transient code 1 Decoder2 w/ K2 Layer 3 decoded Encoded Transient code 2 code, data Transient code 1 Layer Encoded 3 decoded code, data Layer 2 decoded Decoder1 w/ K1 Layer 1 decoded Layer 3 decoded Layer 2 decoded Original memory First snapshot Second snapshot Third snapshot 3

5 Solution: CodeXt Framework built upon S2E Selective means QEMU vs KLEE (LLVM) Decision made per basic block Run-time info of the attack CodeXt Report Run-time memory dump Symbolic Execution Dynamic Binary Analysis Run-time analysis info Offline Analysis Recovered code Obfuscation info Intermediate results 4

6 CodeXt Output Instruction Trace of executed instructions Grouping of fragments into chunks Reveals original and unpacked malcode Assisted by a translation trace Data Trace of memory writes Intelligent memory update clustering Multi-layer snapshots Call Trace of system calls With CPU context 5

7 Problems + Challenges + Solutions 6

8 Handling Byte Streams S2E expects well structured binaries We wrap the binary for execution Info Buffer Output Host to Guest File Transfer Wrapper Guest OS CodeXt S2E Plugin S2E (Modified QEMU) S2E uses basic block granularity Our modified QEMU translation returns more info We leverage translation and execution hooks to verify 7

9 Recognizing Code Avoiding the Halting Problem No infinite loops Caps on executed instructions Different types: target, non-target, system False cognates Illegal first instructions False jumps into suffix Many substrings Matched code fragment: ends on system call, EAX within range Detection Point Random bytes Hidden Code Random bytes Hidden Code Random bytes Hidden Code 8

10 Dealing with Code Fragments Fragmentation Clustering into Chunks, adjacency, execution trace Density Usage: Executed/Range Overlay: Unique executed/range over snapshots Enclosure Continuous executable bytes adjacent to end S2E ( ) S2E (, offset 1 ).. Fragments Match S2E (, offset n ) 9

11 Defeating Obfuscation FPU instructions, fnstenv Added small change to QEMU to comply Intra-basic block self-modification We know address range of each translated block During execution we track writes If any write is to same block we retranslate block Emulator detection Tested for a set of obscure instructions used as canaries 10

12 Results 11

13 Experiments Hidden code search 1KB to 100KB buffers, 40B to 80B shellcodes Filled with either null, live-capture, or random bytes Varied assistance data: EIP, EAX, both, neither Accuracy De-obfuscation, Anti-emulation detection Various packers mentioned in previous research In-shop: Junk code insertion, Ranged xor, Incremental Symbolic Branching 12

14 Multi-Layered Encoders xor_key1 xor_key2 xor_key2 of xor_key1 junk inserted bytes 13

15 Incremental Encoder Decoder3 w/ K3 Decoder3 w/ K3 Decoder3 w/ K3 Decoder3 w/ K3 Encoded code, data Transient code 1 Decoder2 w/ K2 Layer 3 decoded Encoded Transient code 2 code, data Transient code 1 Layer Encoded 3 decoded code, data Layer 2 decoded Layer 1 decoded Layer 3 decoded Layer 2 decoded Decoder1 w/ K1 Original memory First snapshot Second snapshot Third snapshot 14

16 Symbolic Conditionals Logical Start func1() in Hidden Code Frag #1 Hidden Code Frag #2 func3() in Hidden Code Frag #3... y=0; z=1; if (x>==10) y=func3(); else if (x>=0) y=func1(); if (y==0) z=0; if (y==1 && z==0) z=4;... 15

17 Conclusion Emulation is heavy-weight, but Accurate and enables anti-anti-sandbox techniques OS independent Symbolic analysis engine opens avenues Taint propagation and analysis Fuller branch exploration and pruning heuristics CodeXt Accurately pinpoints and models even highly obfuscated code in adverse conditions. 16

18 Current/Future Development Full binary (ELF/PE) support Modeling unmodified executables Without source code Data and code based taint analysis Can mark any input e.g., all network socket reads Follow not only by data, but instruction influence 17

19 Thank you for your time Any questions? Post-talk, please feel free to contact us 18

T Jarkko Turkulainen, F-Secure Corporation

T Jarkko Turkulainen, F-Secure Corporation T-110.6220 2010 Emulators and disassemblers Jarkko Turkulainen, F-Secure Corporation Agenda Disassemblers What is disassembly? What makes up an instruction? How disassemblers work Use of disassembly In