File System Concepts File Allocation Table (FAT) New Technology File System (NTFS) Extended File System (EXT) Master File Table (MFT) 1
FILE SYSTEM CONCEPTS: FILE ALLOCATION TABLE (FAT) Alex Applegate 2
Overview File Allocation Tables The Reserved Area The FAT Area The Data Area exfat 3
File Allocation Tables Method used by early Microsoft Operating Systems to keep track of files Maintains an entry for each file actively installed on the file system FAT-12 commonly used in removable media (most notably floppy disks), FAT-16 used for hard disk storage, later upgraded to FAT-32 A newer version is also being targeted at mobile devices, which is exfat 4
File Allocation Tables FAT-12, FAT-16, and FAT-32 are almost identical with the exception of the size of the entry for each of the files and the number of addressable clusters FAT file systems have 3 primary areas: a reserved area for file system category data, the FAT area which describes the status of each of the clusters, and the Data area which contains file system metadata and file content 5
File Allocation Tables Source: Microsoft FAT Specification 6
FAT Reserved Area The first portion of the file system is also the reserved area and always begins with the boot sector for the file system The boot sector is the only entry in the reserved area for FAT-12 and FAT-16, but it is possible that the reserved area is larger than the boot sector in those file systems 7
FAT Reserved Area Key Fields Size of a sector (almost always 512 bytes) Size of a cluster Size of FAT area (no. of sectors, no. of FATs) Active FAT(s) Size of root directory (only in FAT-12 & FAT-16) Size of file system Size values are raw numbers except in exfat, which uses powers of 2 8
FAT-32 Reserved Area FAT-32 also has a Backup Boot Volume and FSInfo region in the reserved area The Backup Boot Volume is a copy of the boot volume, which is the first 3 sectors of the file system (1536 bytes) The FSInfo region is optional, but its purpose is to improve the efficiency with which the file system finds unused clusters 9
FAT Concepts - The FAT Area Immediately follows the reserved area Must contain at least one allocation table, but can hold more Each allocation table is a bitmap of the clusters used to indicate whether each cluster is in use by the file system or available to be written to 10
FAT Concepts - The Data Area Contains each of the individual clusters available to the file system In FAT-12 and FAT-16, cluster 0 and cluster 1 are the mapping for the root directory In FAT-32 there is no fixed root directory, so the usable data clusters begin immediately after the FAT area In either configuration, the numbering for usable cluster number begins at 2 11
FAT Concepts - exfat exfat does not use a true reserved area in the same way other FAT systems do There is a Primary Boot Region, followed by a Backup Boot Region, then the FAT Region and the Data region Both boot regions together are 24 sectors long Boot Sector Extended Boot Sector OEM Parameters A reserved region A checksum of the entire boot sector 12
exfat Source: SANS Reverse Engineering Microsoft exfat File System 13
Summary File Allocation Tables The Reserved Area The FAT Area The Data Area exfat 14
File System Concepts - FAT QUESTIONS? 15
File System Concepts: NT File System (NTFS) Alex Applegate 16
Overview NT File System (NTFS) NTFS Layout Recycle Bin Artifacts RECYCLER Data $Recycle.Bin Data 17
NT File System Much greater complexity than FAT Comprised a major operating system rewrite not automatically reverse compatible Microsoft has not published a public specification File system-level support for advanced security features, encryption, and file compression Very dynamic management of file system components and their locations 18
NT File System Introduced the Windows Event Log Highly interlaced with the Windows Registry Changed the security model to implement separate kernel and user modes Updated the functionality of the recycle bin Introduced the Master File Table 19
Event Log Viewer (Windows 7) 20
Registry Editor 21
NTFS Layout Boot sector is the first sector in the file system Master File Table (MFT) structure replaced File Allocation Table Can reside anywhere Covered in greater detail in next lecture Allocation Bitmap is used to keep track of which clusters in the file system are in use 22
NTFS Layout MFT has a duplicate copy stored elsewhere in the file system Clusters numbered from the beginning of the Boot sector starting with zero Small files may be stored entirely in MFT Several different versions of NTFS since WindowsNT 3.1, each with small differences NTFS partitions can interface directly with FAT-32 partitions 23
Recycle Bin Artifacts File from WinNT to WinXP/2003 Drive:\RECYCLER\<SID> File for Vista/Windows7 Drive:\$RECYCLE.BIN\<SID> Only a virtual folder for viewing through Windows Stores user that deleted the file, original file name and path, original file size, time and date the file was deleted 24
RECYCLER Data Each file is a separate file in the recycle bin and is renamed to prevent collisions File names follow a standard format D<orig. drive letter><seq. no.>.<orig. extension> The original file name, path, and deletion time are stored in a file named INFO or INFO2 25
$RECYCLE.BIN Data (Vista+) Every file still has an individual file in the recycle bin Deleted files renamed - $R<hash>.<orig. ext.> Data that had been stored in INFO/INFO2 is stored in files matched to the deleted files Name scheme is $I<hash>.<orig. ext> Hash matches the hash for the deleted file 26
Summary NT File System (NTFS) NTFS Layout Recycle Bin Artifacts RECYCLER Data $Recycle.Bin Data 27
File System Concepts - NTFS QUESTIONS? 28
File System Concepts EXT2/3/4 Alex Applegate 29
Overview General Concepts File System Structures Block Size Characteristics Block Groups Inodes Directories Superblocks 30
General Concepts Popular revision of the basic Unix file system Linux BSD Irix Based on a block system Every resource is a file 31
File System Structures Blocks Basic data chunk for data storage Assigned as a part of a larger block group Common Sizes 1 KiB 2 KiB 4 KiB 8 KiB Block 0 is always the beginning of the device and contains the boot record if device is bootable Block Group A cluster of blocks maintained as a unit to minimize seek time when reading large amounts of consecutive data Inode (Index Node) Each object represented by an inode Contain pointers to the file system blocks for the object Superblock Metadata about the configuration of the file system 32
Block Size Characteristics Upper Limits 1KiB 2KiB 4KiB 8KiB file system blocks 2,147,483,647 2,147,483,647 2,147,483,647 2,147,483,647 blocks per block group inodes per block group bytes per block group file system size (real) file system size (Linux) 8,192 16,384 32,768 65,536 8,192 16,384 32,768 65,536 8,388,608 (8MiB) 4,398,046,509,056 (4TiB) 2,199,023,254,528 (2TiB) [a] 33,554,432 (32MiB) 8,796,093,018,112 (8TiB) 8,796,093,018,112 (8TiB) 134,217,728 (128MiB) 17,592,186,036,22 4 (16TiB) 17,592,186,036,22 4 (16TiB) 536,870,912 (512MiB) 35,184,372,080,64 0 (32TiB) 35,184,372,080,64 0 (32TiB) blocks per file 16,843,020 134,217,728 1,074,791,436 8,594,130,956 file size (real) 17,247,252,480 (16GiB) 274,877,906,944 (256GiB) 2,199,023,255,552 (2TiB) 2,199,023,255,552 (2TiB) file size (Linux 2.6.28) 17,247,252,480 (16GiB) 274,877,906,944 (256GiB) 2,199,023,255,552 (2TiB) 2,199,023,255,552 (2TiB) Source: http://www.nongnu.org./ext2-doc/ext2.html 33
34
Block Groups Blocks are clustered into block groups Information regarding the block groups is stored in a descriptor table in the blocks immediately after the superblock First two blocks are reserved for block usage bitmap and inode usage bitmap Just like allocation tables Limited to one block per bitmap Block after the bitmaps is the inode table In most cases, a data block should appear in the same block group as its inode 35
Inodes Central mechanism for the ext file system Every object in the file system maps to an inode Contains pointers to each of the data blocks associated with a file system object Also stores all of the information/metadata about the file system object except its name Pointers are layered First 12 pointers point to the first 12 blocks of data Next pointer is a single indirect pointer (points to pointers to blocks) Next pointer is a double indirect pointer Next pointer is a triple-indirect pointer Inodes are stored in inode tables (limited to one per block group) 36
Inode Layout 37
Inode Layout (2) 38
Directories Directories are a special type of file object Associates file names of subordinate objects with an inode number Different revisions of ext use different data structures to store contained file names Singly-linked list Hashes Binary Trees 39
Superblocks Contains the configuration of the file system Primary superblock is stored at offset 1024 bytes from the beginning of the device Backup copies are stored across the device Formerly at the beginning of every block group Current revisions use superblocks 0, 1, and powers of 3, 5, and 7 Regardless of the block size, the superblock always begins at offset of 1024 bytes Block group 0 always starts with the superblock, which mean that it may or may contain data block 0 40
Unix File System 41
Summary General Concepts File System Structures Block Size Characteristics Block Groups Inodes Directories Superblocks 42
File System Concepts EXT2/3 QUESTIONS? 43
File System Concepts: NTFS Master File Table Alex Applegate 44
Overview NTFS Master File Table (MFT) MFT Attribute Pairs MFT Residence MFT Time Stamps 45
NTFS Master File Table (MFT) Upgraded replacement to the File Allocation Table NTFS system structures are controlled through the dollar sign abstraction $Boot The file system boot sector $MFT The master file table $MFTMirr The backup master file table $Bitmap Cluster allocation bitmap $Recycle.Bin Deleted files still residing in the file system (Vista+) 46
NTFS MFT (cont d) The MFT traditionally resides close to the beginning of the file system The backup MFT traditionally resides near the middle of the partition and is kept should the primary MFT become corrupted MFT stores metadata for files in the system Every file in the file system has an entry in the MFT, including the MFT itself 47
Predefined Master File Table Entries 48
The Master File Table 49
MFT Attribute Pairs Data stored in MFT is in attribute pairs: the type of attribute and the value for those properties The three most common attributes are $STANDARD_INFORMATION, $FILENAME, and $DATA If an alternate data stream is created for a file, then the file will have multiple $DATA attributes 50
Master File Table Layout 51
Master File Table Entry 52
MFT Residence Most entries in the MFT are non-resident (the data is stored in other clusters and pointed to by the MFT) Small files (specifically, attribute values) may be stored in their entirety in the MFT The boundary is estimated at between 700-800 bytes 53
MFT Timestamps Each entry in the MFT has 8 timestamps 4 in $STANDARD_INFO, 4 in $FILENAME Last Modified, Last Accessed, Last Changed, Created Timestamps are 64-bit values Number of nanoseconds since Jan 1, 1601 relative to UTC For investigation, adjustment must be made for local time zone 54
Activities That Affect Timestamps Source: SANS FOR408 Course 55
Summary NTFS Master File Table (MFT) MFT Attribute Pairs MFT Residence MFT Time Stamps 56
File System Concepts: NTFS Master File Table (MFT) QUESTIONS? 57