Introduction. Collecting, Searching and Sorting evidence. File Storage

Transcription

1 Collecting, Searching and Sorting evidence Introduction Recovering data is the first step in analyzing an investigation s data Recent studies: big volume of data Each suspect in a criminal case: 5 hard disks, 140 CDs or DVDs 4 memory cards and USB sticks Business cases Data: 31 hard disks, 14 terabytes for one case FBI s regional computer forensics lab 2013: 5973 TBs of data from 7273 exams 2 Introduction Audit report: 1566 outstanding cases (2015): 57% waited bet 91 days and over 2 years Recent studies: anti-forensics tools delete files, overwrite clusters multiple times, create large volume of data of certain types Discussion: Collecting evidence: file system, file deletion Techniques for recovering files Existing tools and challenges 3 File Storage Files are stored in file system Files: sequence of binary data (bits and bytes) Data is stored in clusters or blocks Blocks corresponding to a file may be Stored contiguously on disk Split and stored all over the disk 4

2 Example: FAT Storage Files: f1.doc, f2.txt, f3.jpg Root table entries Filename Starting block f1.doc 102 f2.txt 106 f3.jpg 110 Block FAT 101 Free EOF EOF Next block 5 Deleted a file entry in the file system is updated to indicate its deleted status clusters that were previously allocated for storing become unallocated and can be reused to store a new file But: data are left on the disk until a new file overwrites them 6 Example: File deletion Example: File deletion Delete f1.doc Root table entries FAT Block Next block 101 Free Delete f1.doc FAT Block Next block 101 Free Filename Starting block?1.doc Free 103 Free 104 Free 102 Free 103 Free 104 Free f2.txt Free 105 Free f3.jpg EOF 108 Free 109 Free Contents of f1.doc have not been deleted 107 EOF 108 Free 109 Free

3 Major types of file structures Example: Contiguous: stored in blocks in a logical order of sequence Fragmented: One or more chunks are not stored in a sequential order (happens when files are added, deleted or modified) Linear (logical order), non-linear Partial files: Incomplete files: some portion of the files are unavailable (overwritten by other data) 9 A: B: C: D: 10 Major types of file structures Major types of file structures Embedded files: Contents of one file are added or stored inside another file: JPEG inside a word document File systems become large Large hard disks: inexpensive, common Huge number of files and fragments Individual files usually lightly fragmented Causes of fragmentation Low disk space Append more data to an existing file 11 Studies: 6% of all files recovered were fragmented Always perform disk allocation which minimizes file fragmentation to reduce seek time and improve file system performance File types of forensic interest (AVI, JPG, ) higher fragmentations than file types of little interest (BMP, TXT, ) JPEG: 16%, AVI: 17% PST: 58% ( , outlook) Word doc: 17% 12

4 Evidence collection Search evidence in the complete file system, including recovering those deleted files File carving: Recovery of file fragments from a digital storage device without the assistance from the file system Scanning the raw bytes of the disk and reassembling them 13 Evidence collection File carving: Possible even if the file system metadata has been completely destroyed Possible even if the files are deleted Delete: means removing the knowledge of where the file is, but not removing the file content Possible to recover files with file name renamed to hidden what the file actually is Possible to recover data that is embedded into another file (JPEG inside a doc) 14 Techniques for File carving Tools for file carving Tools have been developed to automate the process of carving for various file types foremost, scalpel and DataLifter, PhotoRec Specialized forensic tools: EnCase, FTK, X- ways Can be used to extract files from physical memory dumps from mobile devices and from raw network traffic 15 Need to understand how the tools carve files Not a substitute for knowledge Understand limitations of tools 16

5 Techniques for File Carving Header-footer Recover files based on known header Used in EnCase, Foremost, Scalpel File Structure Header-footer + internal layout of a file Use in Foremost, PhotoRec Content-based (Semantics) Header-Footer Carving Most basic carving technique Steps Scan for the header of a file type Once found, scan for the file type s footer File = bytes between header and footer copy byte-by-byte Examples of File Signatures Header Footer File type FFD8FF FFD9 Jpg, jpeg 424D BMP FFFB MP3 without ID3 tag MP3 with ID3 tag D0CF11E0 Doc Wav Pdf B GIF 19 20

6 Header-Footer Carving Problem: Header/footer markers: short May produce many results (false positives) Cannot handle fragmented/partial files Cannot carve files without fixed headers (text/html) File signature: ml Fragmented example Variations Estimate the file size through various means Header-maximum file size carving Fixed the number of bytes in file carving after locating the header Header-embedded file size carving Find out the file size through the information available in the header 23 24

7 Header-Maximum File Size Carving Carve a fixed no of bytes from the beginning of a possible file Steps Scan for the header of a file type Extract a fixed no of bytes Size determined by trial and error Can be useful for files with footers JPEG: store thumbnails within the image Thumbnail: another JPEG Are not affected if additional data is appended to the end of JPEG 25 Header-Maximum File Size Carving Same problem as header-footer carving Always return results much larger than the original file Manual process to discard additional data If the guess for the maximum size is too small carved incorrectly 26 Header-Embedded File Size Carving Many files: embed info about the file size in the first few bytes Find out the size of the file by analyzing the embedded info Steps Scan for the header of a file type Determine the file size by reading the bytes extract Header-Embedded File Size Carving 27 28

8 File Structure Based Carving Carve by using knowledge about the internal file structure Metadata Header, footer, identifier strings, size info, etc Can be used to detect cases of fragmentation if the file structure data is detailed and extensive Example: File structure JPEG file Header Start of image: FF D8 EXIF info Start of image data A series of sections End of image data Footer End of image data (FF D9) File structure PNG file Header byte Size of the next section IHDR: identifier of the next section 12 bytes: unstructured data Challenges in File carving Original file may be fragmented carving process that assumes all portions of the file was stored contiguously on the disk will fail salvaging fragments of multiple files and incorrectly combining them into a single container Content-based carving Main idea: read individual block and analyze its contents to find out if it belongs to a particular file 31 32

9 Content-based Carving Main idea: Fragmentation can occur only at block boundaries Block: size of the smallest data unit that can be written to a storage media (sector or cluster size) One block one single file Information entropy Entropy: measure of randomness Large changes in entropy Indicate that the sector belongs to a different file Entropy Example 1: tossing a coin: Possible outcomes: head/tail Prob(head) = Prob(tail) = N n 1 Entropy = p log p n 2 n 33 Entropy Example 2: In a bin, there are four different colored balls: red, yellow, blue and green. There are 9 red color balls, 1 yellow color ball, 1 blue color ball and 1 green color ball N n 1 Entropy = p log p n 2 n Entropy Example 3: In a bin, there are four different colored balls: red, yellow, blue and green. There are 3 red color balls, 3 yellow color balls, 3 blue color balls and 3 green color balls N n n 1 Entropy = p log p 2 n

10 Sliding entropy Sliding window Measure average value of the bytes Entropy formula: N pn n 1 log N: total number of different values P n : probability of the n-th value 2 p n Sliding entropy Sliding window Measure average value of the bytes Bytes: 8 bits: values = 0 to 255 Entropy: : Text and HTML blocks 7 8: zip and JPEG blocks Studies txt and jpg Studies Mp3 file, zip version, encrypted version 39 40

11 Example Sliding entropy Calculate the entropy of the block of the data If the block contains compressed data entropy of these blocks would be similar If a sudden in entropy that block doesn t belong to PNG image data Example: sliding entropy Block: Block: Example: sliding entropy Remove the section where the entropy drops: 43 44

12 Data inbetween zip files Current Research approach 1 Stage 1: Header/footer Stage 2: 45 Complete JPEG file for segment 2 46 Stage 3: Decoding to RGB Stage 4: Fragmentation point high CED value CED ED ED boundary nearby Current Research approach 1 Stage 6: Aim: construct from header to footer Join segments together Boundary: RGB values of pixels on both sides of the boundary Nearby: RGB values of pixels on one side of the boundary 47 48

13 Current Research approach 2 Current Research approach 2 Graph approach Assume all file clusters are randomized Step 1: identify headers/footers 49 Step 2 For each header, find the best match (using similarity) Similarity calculation would depend on the content of the cluster Image file: check block similarity Text file: check word likelihood 50 Current Research approach 2 Comparison of different methods Probability/likelihood Lots of different tools/methods for file carving Performance comparison: Carving quality Memory and space used Terminology: Positive: a file that is correctly carved from the dataset 51 52

14 Quality Terminology: False positive: a carving result which is not a positive False negative: a file that is present in the dataset, but was not carved In dataset Recovered Yes No Yes Positive False positive No False negative 53 Quality Recall: proportion of the files is recovered tp Recall tp fn Precision: proportion of the recovered files is correct tp Precision tp fp F measure: control user s preference on recall and precision 1 Fmeasure P R Example Consider there is a total of 10,000 files. Out of these 10,000 files, there are 100 files that are fragmented. Suppose that a tool can only recover 60 fragmented files. Determine Recall Precision Accuracy Performance Analysis Public datasets: FAT carving test dataset (15 files) dftt.sourceforge.net/test11 DFRWS 2006 challenge image (32 files) dfrws.org/2006/challenge Basic data carving test: Simple datasets good results Complex datasets poor results Fragmentation of files: major impact 56

15 Tools comparison Look at Percentage of files recovered The correctness and reliability of tool output Processing speed of the tool Requirement: Process roughly 100GB data per day 1.16 MB per second Handle less than 0.58 MB Impractical datasets Tools comparison: example 1 Basic data carving test: Contains Valid doc, jpeg, wav, pdf, zip, gif, doc, xls files Invalid jpg file (header has been modified) Deleted ppt, wmv files Contains only contiguous files E.g., PhotoRec: can find all, except invalid jpg files (bec header info is not correct) Tools comparison: example 1 Tools comparison: example 2 Another test set shtml Contains Jpeg, zip, html, txt, word files Fragmented files 59 60

16 Tools comparison: example 2 Example: One JPEG non-fragmented One JPEG non-fragmented, larger than a typical default max file size One JPEG non-fragmented, but sector before it has 0xffd8 in the first two bytes One JPEG fragmented with text in between One JPEG fragmented with a Word document in between One JPEG fragmented with random data in between 61 Tools comparison : example 2 Example One JPEG fragmented with a JPEG in between Two JPEGs that are intertwined One JPEG non-fragmented that is REALLY big One JPEG fragmented with singe sector in between that starts with 0xffd9 E.g., PhotoRec: Performance drops because the dataset is more complicated Contiguous + fragmented files 62 Tools comparison : example 2 Tools comparison : example

17 General Findings MPEG, ZIP: Difficult to carve bec of common header values Scalpel: header-based carving PhotoRec: structure-based carving Contiguous files: good performance Fragmented: not easy Use of file carving to solve Data hiding conceal a file: change its name to mislead digital investigators Renaming an illegal photograph from xxx.jpg to xxx.exe Need to check the file header (file signature) The file xxx.exe that has a JPEG header (FF D8 FF) will be correctly recognized as a graphics file Steganography? It hides info inside image files Two types: insertion and substitution Insertion Hidden data is not displayed when viewing the original file Need to analyze the data structure carefully Hidden message 67 68

18 Steganography? Substitution Replaces bits with other bits of data Usually change the last two LSBs (least significant bit) Original pixel Altered pixel Steganography Detect variations of the graphic image When applied correctly you cannot detect hidden data in most cases Check to see whether the file size, image quality, or file extensions have changed Clues to look for: Duplicate files with different hash values Steganography programs installed on suspect s drive 70 Data and File carving Summary DeepSound: is a steganography tool and audio converter that hides secret data into audio files. The application also enables you to extract secret files directly from audio files or audio CD tracks. Collecting evidence: file storage clusters fragmentation file deletion Data remains in clusters Techniques for recovering files Header-footer, file structure, content-based approach Existing tools and challenges 72

19 Summary Bec of the large volume of data Investigator: analyze data and understand interrelationships Gold standard: analyze all files to ensure nothing is overlooked Now: intelligence-based : subset of files are analyzed dependent upon the intelligence provided to the investigator Not find every piece of evidence, rather sufficient evidence to determine innocence of guilt Summary File Carving vs Keyword searching: Looks for data that fits into known file structures and interprets that data in light of these structures Search for content that matches one or more keywords or keyword patterns Find structures matching known structures vs Find data matching known data Summary Bec of the large volume of data Investigator: analyze data and understand interrelationships Gold standard: analyze all files to ensure nothing is overlooked Now: intelligence-based : subset of files are analyzed dependent upon the intelligence provided to the investigator Not find every piece of evidence, rather sufficient evidence to determine innocence of guilt 75