Random Sampling applied to Rapid Disk Analysis

Transcription

1 1/28 Random Sampling applied to Rapid Disk Analysis System & Network Engineering Research Project Nicolas Canceill July 4, 2013

2 1 Rapid Disk Analysis 2 The Math 3 The Aftermath 2/28 4 Conclusions

3 2/28 Introduction Background Assoc. Prof. S. Garfinkel Navy Postgraduate School Advanced Forensics Format The Sleuth Kit Better analysis for digital evidence Searching a 1TB hard drive in 10 minutes (ACM 2013) Research E. van Eijk, Z. Geradts Nederlands Forensisch Instituut Stability? Scalability? Precision?

4 3/28 1 Rapid Disk Analysis 2 The Math 3 The Aftermath 4 Conclusions

5 4/28 Rapid Analysis: Why? Traditionally: investigation was leisurely Reading a 1TB hard drive: about 3.5h The cost of seek : 1 36GB 100, KiB New challenges Large installations: computers room, datacenter... Forensics control at checkpoints: border crossing, airports... The bomb will go off in the next hour!

6 5/28 Rapid Analysis: What for? Profit Indications Data analysis Determine free/wiped space Characterize data based on signatures Hash sectors to look for specific data

7 6/28 Rapid Analysis: How? Data characteristics Described (header/trailer) Encoded/formatted Sectorized and distributed Analysis strategies Simplify: hashing Tolerate: extract signature Reduce: random sampling

8 7/28 Research scope Research question How can random sampling help forensically investigate hard disk drives? What kind of indications may be provided? Which parameters are in play? Which degree of certainty may be achieved?

9 8/28 1 Rapid Disk Analysis 2 The Math 3 The Aftermath 4 Conclusions

10 9/28 Analysis process Built on top of S. Garfinkel s frag_find tool Input Image file to search Data-set/Signatures-set to look for Parameters: hashing, sampling, tolerance Process Build Bloom filter (hashing) Select sample For each block in sample: filter (and compare)

11 10/28 Random sampling: Basic model Using a random sample of a statistical population to estimate/predict characteristics Simple scenario Is this hard drive empty/wiped? M empty blocks out of N n sampled blocks out of N Error rate The probability to sample only empty blocks: E = i=n i=1 N (i 1) M N (i 1)

12 11/28 Random sampling: Data layout Data is sectorized: Data is not always aligned:

13 12/28 Random sampling: Advanced model A more realistic scenario Does this hard drive contain the target block? All possible offsets: overlap transactions by B F C All possible transactions: N = T (B F ) All target transactions: M = D T Error rate The probability to miss all target blocks: i=n C (i 1) D T (B F ) T E = (i 1) i=1 C T (B F )

14 13/28 Experimental protocol Experimental image set Parameters: image size, sector size, % of empty sectors, length of target data, offset size Input: Random files and NSRL Reference DataSet Experimental process Parameters: image size, sector size, transaction size, sampling fraction Randomly select a master file signature Generate several images (length of target data, % of empty sectors) Successively run several timed searches

15 14/28 1 Rapid Disk Analysis 2 The Math 3 The Aftermath 4 Conclusions

16 15/28 Results: statistical distribution 0.6 Presence of target data Nb. of transactions

17 16/28 Rapid Disk Analysis The Math The Aftermath Conclusions Results: block-to-transaction scaling Avg. error variance Transaction size 2 blocks 4 blocks 8 blocks Sample size (blocks)

18 17/28 Results: precision scaling Avg. error variance Nb. of transactions Image size 2MB 4MB 10MB 20MB

19 18/28 Results: time scaling Avg. search time (seconds) Image size 200kB 400kB 1MB 2MB 4MB 10MB 20MB 40MB 100MB Nb. of sampled blocks

20 19/28 Results: time overhead Avg. search time (seconds) Nb. of transactions Image size 2MB 4MB 10MB 20MB

21 20/28 1 Rapid Disk Analysis 2 The Math 3 The Aftermath 4 Conclusions

22 21/28 Contributions Main findings Parameters analyzed: Image characteristics: image size, sector size, data alignment, size of target data Sampling settings: sample size, transaction size, tolerance Scalability: Sample size scales with time: S t Error rate scales with time: E 1 t Public material Fork of S. Garfinkel s tools on GitHub Most of experimental scripts on Gist

23 22/28 Research answers What kind of indications may be provided? Presence/absence of target data or signature Which parameters are in play? Disk and data characteristics Sampling parameters Which degree of certainty may be achieved? Certainty scales well with time Insight about target disk will improve certainty Random sampling is a powerful, scalable, adaptive technique for fast HDD analysis Efficiency relies on suitable sampling settings, and limited insight on target HDD

24 23/28 Further research Improving insight of target Pre-determine sector size, data alignment Look for optimal block-to-transaction ratio One step further: pre-sampling Automate decision process Optimal time spending Automatic settings balance Simple user-side: time or certainty

25 24/28 Appendix 1: Bloom Filter (a) Hash-based filtering technique Initialize An array of n bits set to zero k different hash functions uniformly mapping to [0 n] Add an element Apply functions to compute k integers in [0 n] Set k corresponding bits to 1 Query an element Apply functions to compute k integers in [0 n] Check if k corresponding bits are all 1

26 Appendix 1: Bloom Filter (b) Avg. error variance Bloom filter size 8 bits 32bits 25/ Nb. of transactions

27 Appendix 1: Bloom Filter (c) 26/28 Avg. building and search time (seconds) Bloom filter size bits 16 bits bits 30 bits bits 32 bits Nb. of transactions

28 27/28 Appendix 2: Data layout (a) Optimal transaction size depends on sector size Best case: Worst case:

29 28/28 Appendix 2: Data layout (b) Optimal transaction size depends on data layout