File Management. Information Structure 11/5/2013. Why Programmers Need Files

Size: px

Start display at page:

Download "File Management. Information Structure 11/5/2013. Why Programmers Need Files"

Moses Walker
5 years ago
Views:

File Mgr Device Mgr Memory Mgr Process Mgr File Mgr Device Mgr Memory Mgr Process Mgr 11/5/2013 Slide 13-1 Slide 13-2 File Management 13 Fig 13-2: The External View of the File Manager Slide 13-3 Why

1 File Mgr Device Mgr Memory Mgr Process Mgr File Mgr Device Mgr Memory Mgr Process Mgr 11/5/2013 Slide 13-1 Slide 13-2 File Management 13 Fig 13-2: The External View of the File Manager Slide 13-3 Why Programmers Need Files Slide 13-4 mount() write() close() open() lseek() read() Application Program WriteFile() CreateFile() CloseHandle() ReadFile() SetFilePointer() HTML Editor File Manager <head> </head> <body> </body> foo.html <head> </head> <body> </body> Web Browser File Manager UNIX Windows Hardware Persistent storage Shared device Structured information Can be read by any applic Accessibility Protocol File Management File is a named, ordered collection of information The file manager administers the collection by: Storing the information on a device Mapping the block storage to a logical view Allocating/deallocating storage Providing file directories What abstraction should be presented to programmer? Slide 13-5 Information Structure Applications Records Structured Record Files Record-Stream Translation Byte Stream Files Stream- Translation Storage device Slide

2 Byte Stream File Interface Slide 13-7 Low Level Files Slide 13-8 fileid = open(filename) close(fileid) read(fileid, buffer, length) write(fileid, buffer, length) seek(fileid, fileposition) fid = open( filename,); read(fid, buf, buflen); close(fid); int open() {} int close() {} int read() {} int write() {} int seek() {} b 0 b 1 b 2 b i Stream- Translation Storage device response to commands Structured Files Slide 13-9 Record-Oriented Sequential Files Slide Records Logical Record Record- Translation fileid = open(filename) close(fileid) getrecord(fileid, record) putrecord(fileid, record) seek(fileid, position) Record-Oriented Sequential Files Slide Record-Oriented Sequential Files Slide Logical Record Logical Record H byte header k byte logical record H byte header k byte logical record Physical Storage s Fragment 2

3 Electronic Mail Example struct message { /* The mail message */ address to; address from; line subject; address cc; string body; }; struct message *getrecord(void) { struct message *msg; msg = allocate(sizeof(message)); msg->to = getaddress(); msg->from = getaddress(); msg->cc = getaddress(); msg->subject = getline(); msg->body = getstring(); return(msg); } Slide putrecord(struct message *msg) { putaddress(msg->to); putaddress(msg->from); putaddress(msg->cc); putline(msg->subject); putstring(msg->body); } ed Sequential File Suppose we want to directly access records Add an index to the file fileid = open(filename) close(fileid) getrecord(fileid, index) index = putrecord(fileid, record) deleterecord(fileid, index) Slide ed Sequential File (cont) Slide More Abstract Files Slide Application structure Account # i k j index = i index = j index = k Inverted files System index for each datum in the file bases More elaborate indexing mechanism DDL & DML Multimedia storage Records contain radically different types Access methods must be general Implementing Low Level Files Secondary storage device contains: Volume directory (sometimes a root directory for a file system) External file descriptor for each file The file contents Manages blocks Assigns blocks to files (descriptor keeps track) Keeps track of available blocks Maps to/from byte stream Slide Organization Boot Sector Volume Directory Blk 0 Blk 1 Blk k-1 Track 0, Cylinder 0 Blk k Blk k+1 Blk 2k-1 Track 0, Cylinder 1 Blk Blk Blk Track 1, Cylinder 0 Blk Blk Blk Track N-1, Cylinder 0 Blk Blk Blk Track N-1, Cylinder M-1 Slide

4 Low-level File System Architecture Slide File Descriptors Slide b 0 b 1 b 2 b 3 b n-1... Sequential Device Randomly Accessed Device External name Current state Sharable Owner User Locks Protection settings Time of creation Time of last modification Time of last access Reference count Storage device details An open() Operation Locate the on-device (external) file descriptor Extract info needed to read/write file Authenticate that process can access the file Create an internal file descriptor in primary memory Create an entry in a per process open file status table Allocate resources, e.g., buffers, to support file usage Slide File Manager Structures 3 Return a reference to the data structure 2 Keep the state of the processfile session Process-File Session Open File Descriptor External File Descriptor Slide Copy info from external to the open file descriptor Opening a UNIX File Slide Management Slide fid = open( filea, flags); read(fid, buffer, len); 0 stdin 1 stdout 2 stderr 3 Open File Table File structure On-Device File Descriptor Internal File Descriptor The job of selecting & assigning storage blocks to the file For a fixed sized file of k blocks File of length m requires N = m/k blocks Byte b i is stored in block i/k Three basic strategies: Contiguous allocation Linked lists ed allocation 4

5 Contiguous Allocation Maps the N blocks into N contiguous blocks on the secondary storage device Difficult to support dynamic file sizes File descriptor Head position 237 First block 785 Number of blocks 25 Slide Linked Lists Each block contains a header with Number of bytes in the block Pointer to next block s need not be contiguous Files can expand and contract Seeks can be slow First block Slide Head: 417 Byte 4095 Byte 4095 Byte N-1 ed Allocation Slide DOS FAT Files Slide Extract headers and put them in an index Simplify seeks May link indices together (for large files) File Descriptor File Descriptor block Head: 417 Byte Byte 4095 N-1 Byte File Access Table (FAT) mode owner Direct block 0 Direct block 1 Direct block 11 Single indirect Double indirect Triple indirect UNIX Files Slide Unallocated s How should unallocated blocks be managed? Need a data structure to keep track of them Linked list Very large Hard to manage spatial locality status map ( disk map ) Bit per block Easy to identify nearby free blocks Useful for disk recovery Slide

6 Marshalling the Byte Stream Slide Full Buffering Slide Must read at least one buffer ahead on input Must write at least one buffer behind on output Seek flushing the current buffer and finding the correct one to load into memory Inserting/deleting bytes in the interior of the stream Storage devices use block I/O Files place an explicit order on the bytes Therefore, it is possible to predict what is likely to be read after byte i When file is opened, manager reads as many blocks ahead as feasible After a block is logically written, it is queued for writing behind, whenever the disk is available Buffer pool usually variably sized, depending on virtual memory needs Interaction with the device manager and memory manager Directories Slide Directory Structures Slide A set of logically associated files and sub directories File manager provides set of controls: enumerate copy rename delete traverse etc. How should files be organized within directory? Flat name space All files appear in a single directory Hierarchical name space Directory contains files and subdirectories Each file/directory appears as an entry in exactly one other directory -- a tree Popular variant: All directories form a tree, but a file can have multiple parents. Directory Implementation Device Directory A device can contain a collection of files Easier to manage if there is a root for every file on the device -- the device root directory File Directory Typical implementations have directories implemented as a file with a special format Entries in a file directory are handles for other files (which can be files or subdirectories) Slide UNIX mount Command / / bin usr etc foo bin usr etc foo / bill nutt bill nutt FS abc cde xyz / FS blah abc cde xyz blah mount FS at foo Slide

VFS-based File Manager Slide 13-37 Exports OS-specific API File System Independent Part of File Manager Virtual File System Switch Rensselaer Polytechnic Institute CSCI-4210 Operating Systems David

MS-DOS Part of File Manager ISO 9660 Part of File Manager ext2 Part of File Manager very fast managed as part of the filesystem volatile non-volatile very small A File Management System is a set of

should be organized and available in a convenient and efficient manner Files are the basic building blocks very slow very large A file is an abstraction that represents user data, OS data, an

7 VFS-based File Manager Slide Exports OS-specific API File System Independent Part of File Manager Virtual File System Switch Rensselaer Polytechnic Institute CSCI-4210 Operating Systems David Goldschmidt, Ph.D. MS-DOS Part of File Manager ISO 9660 Part of File Manager ext2 Part of File Manager very fast managed as part of the filesystem volatile non-volatile very small A File Management System is a set of OS services that supports files and directories for user applications, OS programs, etc. should be organized and available in a convenient and efficient manner Files are the basic building blocks very slow very large A file is an abstraction that represents user data, OS data, an executable, a device, etc. A file is simply a sequence of bytes Actual storage location (via network?) and format are transparent to users Storage scheme on disk is also transparent Typically involves cylinders, tracks, sectors, etc. File attributes include: Human-readable symbolic name Type (e.g. executable, directory, text, PDF, etc.) Logical location (i.e. containing directory or path) Physical location on disk Size (in bytes) Protection or security (i.e. permissions) Timestamps (created, last modified, last accessed) 7

The pathname (or just path) of a file specifies the sequence of directories one must traverse to locate the file An absolute path starts at the root node A relative path starts anywhere A link

shortcut is just a symbolic link File creation requires space allocation Opening a file returns a handle or file descriptor Read and write operations use the handle and an offset (or file pointer)

8 The pathname (or just path) of a file specifies the sequence of directories one must traverse to locate the file An absolute path starts at the root node A relative path starts anywhere A link provides a shortcut to a file and may circumvent the given directory hierarchy A hard link in Unix is indistinguishable from the original file A symbolic link in Unix is merely a shortcut A Windows shortcut is just a symbolic link File creation requires space allocation Opening a file returns a handle or file descriptor Read and write operations use the handle and an offset (or file pointer) The close operation simply deletes the handle and deallocates any memory Deleting a file deallocates all disk space marked as in use by the file But likely does not erase file contents Deleted files are recoverable until the disk space is used for (and overwritten by) other file(s) The delete operation also removes the corresponding entry in the containing directory An access method describes the manner and mechanisms by which a process accesses the data in a file In a contiguous disk space allocation scheme, files are allocated to contiguous blocks of disk space Two common access methods: Sequential access (open, read, write, close) Random access (open, read, write, seek, close) 8

Four files allocated contiguously to disk: A A A B B C C C D File B outgrows its space and is

Defragmentation combines free disk space: A A A C C C B B B B D D In a ed disk space allocation scheme,

A A B B C C C D File B outgrows its space, so one or more new s are allocated: A A A B B C C C D B B File

boundaries, causing slower response times Using a linked approach, each has a pointer to the next Using

s) may follow the same scheme as for a single file Linked scheme ed approach Bitmapped approach: A bitmap

9 Four files allocated contiguously to disk: A A A B B C C C D File B outgrows its space and is reallocated: A A A C C C D B B B B File D outgrows its space and is reallocated: A A A C C C B B B B D D Defragmentation combines free disk space: A A A C C C B B B B D D In a ed disk space allocation scheme, files are allocated to s of disk space on an as needed basis Four files allocated contiguously to disk: A A A B B C C C D File B outgrows its space, so one or more new s are allocated: A A A B B C C C D B B File D outgrows its space, so one or more new s are allocated: A A A B B C C C D B B D File accesses may cross boundaries, causing slower response times Using a linked approach, each has a pointer to the next Using an indexed approach, a single table maintains pointers to each individual Managing free blocks (or free s) may follow the same scheme as for a single file Linked scheme ed approach Bitmapped approach: A bitmap is maintained in memory Each bit corresponds to a block or A 0 indicates a free block A 1 indicates the block is in use 9

A file allocation table (FAT) maps logical files to their physical addresses on a disk A pure FAT would have an entry for each sector (e.g. 512 bytes) To improve performance, s of contiguous sectors are used Clusters range from 4 to 64 sectors (e.

contains: Mode and type of file Number of links to the file File owner s userid and groupid Size of file (in bytes) Last access and last modified times Number of blocks allocated to the file Pointers

10 A file allocation table (FAT) maps logical files to their physical addresses on a disk A pure FAT would have an entry for each sector (e.g. 512 bytes) To improve performance, s of contiguous sectors are used Clusters range from 4 to 64 sectors (e.g. 2,048 to 32,768 bytes) Microsoft FAT-32 filesystem maps to s May require linked s to house entire FAT Linux uses indexed s In Unix, file information is stored in an index node (I-node), which contains: Mode and type of file Number of links to the file File owner s userid and groupid Size of file (in bytes) Last access and last modified times Number of blocks allocated to the file Pointers to the first twelve blocks Pointers to three additional blocks of pointers triple indirect single indirect 1024 double indirect A virtual filesystem provides transparent access to different filesystem types on multiple device types and disk partitions 10

A virtual machine is a layered approach that logically combines the kernel operating

virtual processor with its own virtual memory Non-virtual machine Virtual machine

command=viewarticlebasic&articleid=9002552 Java programs execute on a native Java Virtual

virtual machine is isolated from all other virtual machines which prohibits direct sharing

11 A virtual machine is a layered approach that logically combines the kernel operating system and hardware Creates the illusion of multiple processes, each executing on its own virtual processor with its own virtual memory Non-virtual machine Virtual machine Java programs execute on a native Java Virtual Machine (JVM) Virtual machines provide complete protection of system resources Each virtual machine is isolated from all other virtual machines which prohibits direct sharing of system resources Virtual machines can be difficult to implement due to the effort required to provide an exact duplicate of each underlying machine 11

CSE325 Principles of Operating Systems. File Systems. David P. Duggan. March 21, 2013

CSE325 Principles of Operating Systems File Systems David P. Duggan dduggan@sandia.gov March 21, 2013 External View of File Manager Application Program mount() write() close() open() lseek() read() WriteFile()