Cross Drive Analysis: A New Analytic Approach for Massive Data Sets

Size: px

Start display at page:

Download "Cross Drive Analysis: A New Analytic Approach for Massive Data Sets"

Stephanie Johnston
5 years ago
Views:

1 Cross Drive Analysis: A New Analytic Approach for Massive Data Sets Simson L. Garfinkel, Ph.D. Consulting Scientist, Basis Technology Thursday, June 7, 2007 Basis Technology Government Users Conference Washington, DC

Computer Forensics Today: Tools for finding hidden data.

# $ % & % ' ( ) ' * & +, - % &. $ ' ' 1 3 4 5 ' 2 ( " % 6 #. & " 5 ' 76",'89:';<<;' 3!5,',%=;' & " '. /, '!.. ( - ", $ ' 0 ( - 1 ) ( - 2, ' '!! "#$!%$&'()*$+)!,-..$(,!.(/*!'+!"#$%&'($)&!*(+$#!,&-.

!"#$!%$&'()*$+)!3/$,! +/)!$*&#',1=$!4'($$(!3$<$6/&*$+)5!'+3!)//6,!./(!&$(./(*'+4$!'&&('1,'6!'($!3$.141$+)0!!>,!'! ($,-6)5!'))/(+$7,!41)$!&//(!?&$/&6$!*'+'8$*$+)@!A7!,-&$(<1,/(,0!! 2&/)".#!3*"&1-!$,&!

3$<$6/&*$+)0!!"#$!C$4)1/+!D#1$.!2/(;./(4$!1,!+/)!31<$(,$!'+3!)-(+/<$(!1,! 6/20!!"#1,!&'))$(+5!4/*A1+$3!21)#!)#$!8$+$('667!6/2!'))$+)1/+!)#')!)#$,$!*'+'8$(,!&'7!)/!,)'..! 4'($$(!3$<$6/&*$+)5!6$'3,!

2 Computer Forensics Today: Tools for finding hidden data. Recover deleted files Detect child pornography Search archives Reconstruct timelines June 2007 S M T W T F S ! "! # $ % & % ' ( ) ' * & +, - % &. $ ' ' ' 2 ( " % 6 #. & " 5 ' 76",'89:';<<;' 3!5,',%=;' & " '. /, '!.. ( - ", $ ' 0 ( - 1 ) ( - 2, ' '!! "#$!%$&'()*$+)!,-..$(,!.(/*!'+!"#$%&'($)&!*(+$#!,&-.(,/&-!+$#$0&+&#)!"#1,$-),(/)(,&0!! "#$($!1,!213$,&($'3!&$(4$&)1/+5!$,&$41'667!'*/+8!*1+/(1)1$,5!)#')!9:!&('4)14$,!6'4;! )('+,&'($+470!!"#1,!($,-6),!1+!'))/(+$7,!&$(4$1<1+8!)#')!&('4)14$,!'($!-+.'1(0!!"#$!%$&'()*$+)!3/$,! +/)!$*&#',1=$!4'($$(!3$<$6/&*$+)5!'+3!)//6,!./(!&$(./(*'+4$!'&&('1,'6!'($!3$.141$+)0!!>,!'! ($,-6)5!'))/(+$7,!41)$!&//(!?&$/&6$!*'+'8$*$+)@!A7!,-&$(<1,/(,0!! 2&/)".#!3*"&1-!$,&!$#!&4),&+&56!/,")"/$5!&5&+&#)!/.!)#$!%$&'()*$+)B,!31<$(,1)7!461*')$0!!"#$7! #'<$!, '+)!'-)#/(1)7!1+!($4(-1)*$+)5!#1(1+85!&(/*/)1/+5!&$(./(*'+4$!'&&('1,'65!4',$! ',,18+*$+)5!'+3!4'($$(!3$<$6/&*$+)0!!"#$!C$4)1/+!D#1$.!2/(;./(4$!1,!+/)!31<$(,$!'+3!)-(+/<$(!1,! 6/20!!"#1,!&'))$(+5!4/*A1+$3!21)#!)#$!8$+$('667!6/2!'))$+)1/+!)#')!)#$,$!*'+'8$(,!&'7!)/!,)'..! 4'($$(!3$<$6/&*$+)5!6$'3,!*1+/(1)1$,!)/!&$(4$1<$!'!6'4;!/.!'3<'+4$*$+)!/&&/()-+1)1$,0!! "#$!%$&'()*$+)B,!'))/(+$7!2/(;./(4$!1,!+.,&!%"7&,-&!)*$#!)*&!8929!5&0$5!:.,;1.,/&E!!FGH!.$*'6$5!4/*&'($3!)/!FIH!1+!)#$!J0C0!6$8'6!6'A/(!&//65!'+3!KLH!*1+/(1)75!4/*&'($3!)/!KMH!1+! )#$!6'A/(!&//60!!"#$!%$&'()*$+)B,!'))/(+$7!2/(;./(4$!1,!'A/-)!$-!%"7&,-&!$-!)*&!1&%&,$5! 0.7&,#+&#)!5&0$5!:.,;1.,/&5!2#/,$!'))/(+$7,!'($!FGH!.$*'6$!'+3!KNH!*1+/(1)70!! <","#0!"-!-&,7"#0!).!+$;&!)*&!=&>$,)+&#)!&7&#!+.,&!%"7&,-&E!!#1($,!1+!MIIK!2$($!OIH!.$*'6$! '+3!MKH!*1+/(1)70!!P+!&'()14-6'(5!)*&!?)).,#&6!@&#&,$5A-!<.#.,-!B,.0,$+!"-!$#!"+>.,)$#)! )..5!./(!1+4($',1+8!31<$(,1)70!!9/+/(,!Q(/8('*!#1($,!1+!MIIK!2$($!NFH!.$*'6$5!4/*&'($3!)/!OLH! /.!)#$!6'2!,4#//6!8('3-')1+8!46',,5!'+3!FIH!*1+/(1)75!4/*&'($3!)/!MKH!/.!)#$!46',,!/.!MIIK0!! R1+/(1)1$,!'($!-"0#"1"/$#)56!(#%&,C,&>,&-&#)&%!"#!+$#$0&+&#)!,$#;-0!!"#$7!4/*&(1,$!/+67! SH!/.!T4'($$(U!CVC!'))/(+$7,!'+3!KKH!/.!,-&$(<1,/(7!>,,1,)'+)!J0C0!>))/(+$7,0!!W/*$+! 4/+,)1)-)$!FKH!/.!CVC,!'+3!FSH!/.!,-&$(<1,/(7!>JC>,0!!>*/+8!XCYKL!'))/(+$7,!1+!)#$! Z1)18')1+8!%1<1,1/+,5!*1+/(1)1$,!4/*&(1,$!KKH!/.!+/+Y,-&$(<1,/(,!'+3!NH!/.!,-&$(<1,/(,5!'+3! 2/*$+!4/*&(1,$!FSH!/.!+/+Y,-&$(<1,/(,!'+3!FFH!/.!,-&$(<1,/(,0!! D"#.,")"&-!$,&!-(E-)$#)"$556!+.,&!5";&56!).!5&$7&!)*&!=&>$,)+&#)!)*$#!:*")&-0!!P+!MIIK5!)#$! '))(1)1/+!(')$!2',!O[H!#18#$(!'*/+8!*1+/(1)1$,!)#'+!2#1)$,0!!"#$($!2',!+/!31..$($+4$!1+!($4$+)! '))(1)1/+!A$)2$$+!*$+!'+3!2/*$+0!! "#$($!'($!'6,/!,)')1,)14'667!-"0#"1"/$#)!,$/&!$#%F.,!0&#%&,!&11&/)-!/+!'!+-*A$(!/.!9:!/-)4/*$,5! !,)'()1+8!8('3$5!4-(($+)!8('3$5!&(/*/)1/+,5!'+3!4/*&$+,')1/+0!!\/(!$]'*&6$5!)#$!'<$('8$! *1+/(1)7!XC!'))/(+$7!1,!4-(($+)67!I0O!,)$&,!6/2$(!)#'+!)#$!'<$('8$!2#1)$5!'+3!)#$!'<$('8$!2/*'+! 1,!I0F!,)$&,!6/2$(!)#'+!)#$!'<$('8$!*'+5!4/+)(/661+8!./(!,$+1/(1)75!8('3$5!'+3!4/*&/+$+)0!! ^',$3!/+!)#$,$! ,5!2$!($4/**$+3!)#')!)#$!%$&'()*$+)!)';$!)#$!./66/21+8!'4)1/+,E!! G4&,/"-&!?@C!$#%!=?@C5&7&5!5&$%&,-*">!)/!,)($,,!)#$!1*&/()'+4$!/.!31<$(,1)7!'+3!)#$1(! 4/**1)*$+)!)/!1)0!!Q-A61467!4/**1)!)#$!%$&'()*$+)!)/!&'(1)7!A/)#!1+!31<$(,1)7!/-)4/*$,!T$0805! 4/*&'('A6$!($&($,$+)')1/+!')!'66!6$<$6,U!'+3!1+!'))1)-3$,!T$0805!_/A!,')1,.'4)1/+U!'*/+8!'66! 3$*/8('&#14!8(/-&,0!!P3$+)1.7!6$<$(,!./(!4#'+8$5!./4-,1+8!/+!>>X,!T2#/!'($!31<$(,$U!'+3! C$4)1/+!D#1$.,0!!P*&6$*$+)!)('1+1+8!/.!6$'3$(,!)/!13$+)1.7!)#$1(!(/6$!1+!,#'&1+8!2/(;!461*')$! 1,,-$,!'+3!1+!$..$4)-')1+8!4#'+8$0!!!

3 Today s tools follow the Rule Of One

4 Today s tools follow the Rule Of One One Drive

5 Today s tools follow the Rule Of One One Drive One Investigator

6 Today s tools follow the Rule Of One One Drive One Investigator One Program

7 Today s tools follow the Rule Of One One Drive One Investigator One Program One Result

8 Drives have no use when the investigation is over.

9 Drives have no use when the investigation is over. Bad guy goes to prison

10 Drives have no use when the investigation is over. Bad guy goes to prison Hard drive goes to storage

11 Today s tools fail when confronted with hundreds of drives. Where do you start?

12 Today s tools fail when confronted with hundreds of drives. Where do you start? Which of these drives are connected?

13 Cross Drive Analysis is a new approach CDA automatically identifies patterns and relationship in data by correlating pseudounique information.

14 Cross Drive Analysis is a new approach CDA automatically identifies patterns and relationship in data by correlating pseudounique information.

15 Cross Drive Analysis is a new approach CDA automatically identifies patterns and relationship in data by correlating pseudounique information. Most Important Drives

16 Cross Drive Analysis is a new approach CDA automatically identifies patterns and relationship in data by correlating pseudounique information. Most Important Drives Previously Unknown Social Network

17 Cross Drive Analysis is a new approach CDA automatically identifies patterns and relationship in data by correlating pseudounique information. Most Important Drives Previously Unknown Social Network

18 Cross Drive Analysis is a new approach CDA automatically identifies patterns and relationship in data by correlating pseudounique information. Most Important Drives Previously Unknown Social Network Probable Network Member

19 Pseudounique Information: rare patterns that mean something Credit card numbers addresses Names Physical Addresses Document fingerprints Disk sector fingerprints Adobe Acrobat Files Blocks of encrypted data Encryption keys etc...

20 The Drives Project started in I purchased this computer for $10 from a local computer store.... it had been a law firm s file server.

21 The computer s hard drive had not been wiped. The hard drive had: Wills Contracts Bills Correspondence More...

22 I started buying lots of used hard drives Drives purchased on ebay & at used computer stores Each disk imaged.... and then I had 250 drives images stored on a RAID array.

Design Principles and Patterns for Computer Systems That Are Simultaneously Secure and Usable by Simson L. Garfinkel S.B., Massachusetts Institute of Technology (1987) S.B., Massachusetts Institute of Technology (1987) S.B., Massachusetts Institute of Technology (1987) M.

23 Design Principles and Patterns for Computer Systems That Are Simultaneously Secure and Usable by Simson L. Garfinkel S.B., Massachusetts Institute of Technology (1987) S.B., Massachusetts Institute of Technology (1987) S.B., Massachusetts Institute of Technology (1987) M.S., Columbia University (1988) Submitted to the Department of Electrical Engineering and Computer Science in partial fulfillment of the requirements for the degree of Doctor of Philosophy in Computer Science and Engineering at the MASSACHUSETTS INSTITUTE OF TECHNOLOGY May 2005 c Simson L. Garfinkel, MMV. All rights reserved. The author hereby grants to MIT permission to reproduce and distribute publicly paper and electronic copies of this thesis document in whole or in part. Author Department of Electrical Engineering and Computer Science May 16, 2005 Certified by David D. Clark Senior Research Scientist Thesis Supervisor Certified by Robert C. Miller Assistant Professor Thesis Supervisor Accepted by A. C. Smith Professor Chair, Committee on Graduate Students Until 2005, we studied sanitization practices. Then we started working on cross drive analysis. Drives in Corpus at Year s End 1,500 1,

24 We discovered cross drive analysis because we were looking for credit card numbers... We were interested in showing that sensitive information had been left behind on hard drives. We wrote a Credit Card Number detector offset 554,553,664 CCN Detector offset 43,332,334, offset 6,334,877,809 Bulk Data...

25 CCN Detector: written in flex and C++ Disk #105: Test # pass pattern 3857 Known prefixes 90 CCV1 43 patterns & histogram 38 Sample output: 'CHASE NA 'DISCOVER 'GE CARD BANK ONE 'NORWEST 'SNB CARD pos= pos= pos= pos= pos= pos=

26 Most drives had just a few, but some drives had a lot of credit card numbers. 40, , 000 Unique CCNs Total CCNs 20, ,

27 Six drives had more than 400 credit card numbers: 40, , , , 000 Unique CCNs Total CCNs Drive # CCNS 1356 unique Drive # CCNS 286 unique Drive # CCNS unique Drive # CCNS 827 unique Drive # CCNS 498 unique Drive # CCNS 223 unique Drive # CCNS 81 unique 200 0

28 Six drives had more than 400 credit card numbers: 40, , , , 000 Unique CCNs Total CCNs Drive # CCNS 1356 unique Drive # CCNS 286 unique Drive # CCNS unique Drive # CCNS 827 unique Drive # CCNS 498 unique Drive # CCNS 223 unique Drive # CCNS 81 unique Supermarket Software Vendor ATM Medical Center Auto Dealership State Secretary's Office

29 We traced 20 drives back to their former owners. While tracing back the drives, we discovered CDA Drive Attribution.

30 Drive Attribution is statistical technique for identifying the former owner of a disk drive. Applications for Drive Attribution: Re-identifying captured drives Return of stolen property Recovering from clerical errors Hard drives usually aren t labeled with their owner s name!

31 Drive Attribution is statistical technique for identifying the former owner of a disk drive. Applications for Drive Attribution: Re-identifying captured drives Return of stolen property Recovering from clerical errors Hard drives usually aren t labeled with their owner s name!

32 What would it mean if two drives had a lot of credit card numbers in common? 10,000 CCNs 500 CCNs 6,000 CCNs 400 CCNs 300 CCNs CCN1 CCN2 CCN3 CCN4 CCN5 CCN6... CCN1 CCN2 CCN3 CCN4 CCN5 CCN CCNs in common!

33 Perhaps the owner of one drive sent an with CCNs to the other drive. CCN1 CCN2 CCN3 CCN4 CCN5 CCN6... CCN1 CCN2 CCN3 CCN4 CCN5 CCN6...

34 Perhaps both owners received an message from a third party. CCN1 CCN2 CCN3 CCN4 CCN5 CCN6... CCN1 CCN2 CCN3 CCN4 CCN5 CCN6...

35 Cross Drive Analysis (CDA) computes the correlation matrix of the pseudounique information. first drive second drive

first drive 2 0 0 4 0 second drive 1 1 3 35 0 3 1

36 Cross Drive Analysis (CDA) computes the correlation matrix of the pseudounique information. first drive second drive High correlation indicates likely extrinsic relationship

37 Here is a correlation of CCNs on the first 250 drives:

38 Here is a correlation of CCNs on the first 250 drives: Same Community College Drives #74 x #77 25 CCNS in common Drives #171 & # CCNS in common Same Medical Center Same Car Dealership Drives #179 & # CCNS in common

39 Used hard drives are a laboratory model. With used drives from America I don t have to worry about the language issue...

40 Used hard drives are a laboratory model. With used drives from America I don t have to worry about the language issue...

41 Initial experience with Cross Drive Analysis. CDA requires pseudo-unique features: CCNs and other financial information Transliterated names addresses, Message-IDs Sector fingerprints Encrypted data Same Community College Drives #74 x #77 25 CCNS in common Same Car Dealership Drives #179 & #206 Drives #171 & # CCNS 13 CCNS in common in common Same Medical Center CDA Process: 1. Extract features 2. Correlate O(n 2 ) 3. Generate Report

42 Cross Drive Analysis: Initial Results Feature Extraction: I/O bound with raw disk images CPU bound with AFF-compressed images (50% better than EnCase) 1-6 GB/hour on a 1.5Ghz 64Bit AMD Athlon Can be run in parallel Took 3 weeks to extract 400 drives Can be done incrementally, as new drives are acquired

43 Cross Drive Analysis: Initial Results Cross Drive Correlation is a memory-bound problem. Tests with 750 drives, 5.8M features, on 1.5Ghz machine with 2GB of RAM: First Implementation: Python Python process grew to 2GB with 100% CPU utilization Python process continued to grow with 1% CPU utilization. Never finished Second Implementation: Highly optimized C++ Process grew to 700MB, then correlation started. Correlation finished in 30 minutes

44 Cross Drive Analysis: Next Generation We have designed a system that uses scalable storage and computation. Cost for analyzing 10,000 drives: $50,000 (guess?)

45 New research: Using Bloom Filters for Cross Drive Analysis A123 B544 B655 C332 Bloom Filter Loading the Bloom Filter Probing the Bloom Filter A123? Bloom YES K552? Filter NO

46 A practical CDA system could have a Bloom Filter for each organization of interest * New information could be automatically screened as it is acquired.

47 Cross Drive Analysis: Status Research Today. Integrate into other forensic applications: Odyssey Plug-in for other vendors Offer CDA as a stand-alone facility Sell CDA as a service. Target customers: Intelligence Law Enforcement Internal Investigations... anyone with more than 100 drives.

Trouble with transactions. Evan Jones

Trouble with transactions. Evan Jones Trouble with transactions Evan Jones http://evanjones.ca A love story A short love story I Afell in short love story love with transactions with transactions Transactions = correct programs A four year