File Management. Logical Structure. Positioning Fields and Records. Primary and Secondary Keys. Sequential and Direct Access.

Transcription

1 File Management Logical Structure Positioning Fields and Records Primary and Secondary Keys Sequential and Direct Access Binary Search File Management File Indexing Chapter 2

2 Logical Structure File, on disk, is simply a collection of bytes Structure must be imposed to organize file logically Field. A single (indivisible) data item Array. A collection of equivalent fields Record. A collection of different fields Logically, file is a stream of bytes representing logical entities For simplicity, we ll assume files are a collection of records 2

3 Fields, Arrays, Records 3

4 Positioning Fields Cannot simply append fields to file, lose their start and end E.g., last_name = Solomon first_name = Mark SolomonMark Must include information to identify field position and contents Fixed length. Fields are fixed to constant size Length indicator. Field begins with its length Delimiter. Separate each field with delimiter character Key value pair. Use keyword=value to represent field, delimiter is also needed to separate key value pairs 4

5 Field Position Advantages/Disadvantages Method Advantages Disadvantages fixed-length simple supports efficient access may be too small may be too large length indicator fields fit data space needed for count delimiter fields fit data space needed for delimiter delimiter must be unique key value efficient if many fields empty space needed for keyword delimiter needed between keywords 5

6 Record Position Records have similar requirements to identify start and end Must include information to identify record position and contents Fixed length. Fields are fixed to constant size Length indicator. Field begins with its length Delimiter. Separate each field with delimiter character Field count. Record begins with number of fields it contains External index. Use external index to track record locations 6

7 Record Position Advantages/Disadvantages Method Advantages Disadvantages fixed-length field count length indicator delimiter simple supports efficient access record fits fields record fits fields record fits fields may be too small may be too large space needed for count variable length space needed for length variable length space needed for delimiter delimiter must be unique variable length external index supports efficient access indirection needed thru index file 7

8 Primary Key How can we identify records during search? Primary key A field (or collection of fields) that uniquely distinguishes records Usually primary key read data field(s) Must be guaranteed to be unique per record Cannot change primary key value(s) without expensive index updates Instead, use system-generated non-data field as primary key 8

9 Secondary Key Non-unique data field Subdivide records into logical groups with a common secondary key value E.g., student major to subdivide student database Assume secondary key groups are commonly required Secondary keys make it computationally efficient to group 9

10 Sequential Access Two ways to access file: sequential or direct Sequential. File elements read in sequence one after another Direct. Elements read directly, with no systematic pattern of access Sequential reads file from start to end Supports sequential, or serial, search Best case, target is first element, O 1 Worse case, target is last element or not in file, O n Average case, target in file, O 1 Τ2 = O n 10

11 Sequential Access on Disk Linear search on external storage much slower Significantly improve absolute performance by reducing seeks Seeks are much more expensive than in-memory operations Can approximate algorithm performance by seeks (disk accesses) Perform record blocking 1. Read m records into memory, search for target 2. Discard if not found, read next m records into memory 3. Continue until search terminates Worse case, O n Τm = O n 11

12 Sequential Search Improvements If we know types of searches, may be able to improve sequential search performance Self-organizing, reorganize order of records to make future searches faster Move to front. When target found, move to front of file Moves common records near front over time Transpose. When target found, move forward one position Bubbles common records towards front over time Count. Sort records by count of times they are requested Keeps common records at the front 12

13 In-Memory Direct Access Jump directly to location of target, read it Single seek, O 1 performance Must be able to convert target primary key into its location Programming example: array indexing Array is a collection of identical records Element index is its key, can be directly converted to address int a[ 256 ] a[ 128 ] ( a ) &a + ( 128 sizeof( a ) ) 13

14 On-Disk Direct Access Want to perform analogous direct-access on file of records 1. Required fixed-length records To calculate how far to offset from front of file for i-th record 2. Require method to convert primary key into offset location Both requirements are non-trivial to support 14

15 Binary Search Perform binary search to improve performance to O lg 2 n 1. File must be sorted, maintaining this proper is expensive 2. Records must be fixed-length 3. Binary search still requires more than 1 2 seeks to find a records Is it worth incurring these costs? If file doesn t change after creation, and often searched, maybe Classic tradeoff between initial cost of construction versus savings after construction 15

16 O n versus O lg 2 n Method n Linear Binary Speedup

17 Sequential Search Scenarios In spite of poor efficiency, sequential search is appropriate in certain situations 1. Searching for patterns 2. Search a file with only a few records 3. Managing a file that rarely needs to be searched 4. Performing secondary key search with expectation of many matches Key tradeoff: cost of searching versus cost of building data structure to support efficient search Above scenarios, building an efficient search structure wasteful 17

18 File Management Files are not static, content changes over their lifetime How can we deal efficiently with addition, update, deletion? Addition. Store data at first available position, or end of file Update. Deletion followed by addition Given this view, our only concern is how to efficiently delete and maintain file structure for follow-on additions 18

19 Record Deletion Record deletion imposes requirements on file management Record initially marked for deletion, but not removed Fixed versus variable-length records complicates strategy Secondary index needed for efficient variable length record deletion One very simply strategy is storage compaction At some point, reclaim storage for all marked records Delay until file closed, until sufficient deletions have occurred Compaction is expensive High availability files may never be free for compaction 19

20 Fixed-Length Deletion Dynamically reclaim space when new records added mark record as deleted, record s space is known as a hole rapidly find previously used space, reallocate to new records Stack of holes, maintain stack of deleted record positions Only works because records fixed-length, can fit in any hole Stack must be persistent as file is closed and re-opened One option, append head of hole stack to end of file Whenever a record is deleted Mark record s space as a new hole Store within new hole current head-of-stack offset Update head-of-stack offset to point to new hole 20

21 Hole Stack (1) Add A,B,C,D; (2) delete B,D; (3) add X,Y,Z Head-of-stack offset set to -1 A,B,C,D added, no holes, so appended B deleted, set hole to head-of-stack (-1), set head-of-stack to B s pos (20) D deleted, set hole to head of stack (20), set head-of-stack to D s pos (60) X added, place at head-of-stack (60), update head-of-stack to 20 Y added, place at head-of-stack (20), update head-of-stack to -1 Z added, no holes, so append 21

22 Variable-Length Deletion More complicated to support dynamic deletion and reallocation Main issue, new records don t exactly fit existing holes To delete, steps are similar to fixed-length records mark record as deleted, add hole to availability list Availability list similar to stack of holes Stores both hole s offset and its size 22

23 First Fit Reallocation Walk availability list for first hole large enough for new record If fit isn t exact, two strategies Pad. Pad record to exactly fit hole Reduces external fragmentation between records Increases internal fragmentation within record Split. Divide hole into new record and remaining fragment Reduces internal fragmentation Increase external fragmentation May need to compact small holes 23

24 Best Fit Reallocation Maintain availability list in ascending order of hole size Now, first fit always choose hole that best fits new record Additional cost incurred to maintain availability list in sorted order Leads to higher cost to find hole for new record Each add can leave a small hole as a fragment Small holes put at front of availability list Need to walk farther to find suitably sized hole for new records 24

25 Worst Fit Reallocation Maintain availability list in descending order of hole size Now, first fit always chooses hole at front of list, if large enough Attempting to produce largest possible fragment on addition Worst fit reduces search for hole to O 1 Eventually, remaining holes will be too small for new records 25

26 File Indexing Many advantages to fixed-length records Direct access, simplicity However, serious drawbacks in terms of space utilization File indexing separates storage from access and management Performed through the use of secondary file index Allows use of fixed-length index file to access data file storing variable-length records 26

27 Simple Indices Simple index Array of key offset pairs that index location of record with primary key k and offset position p in data file Simple indices support indirection Records in the data file can be rearranged by rearranging indices that reference those records, without touching the data file More efficient Access variable-length records through fixed-length index file Pinned records records that cannot change position are supported 27

28 Music File Example Consider a data file containing variable-length records describing music files Primary key: combination of distribution company and recording ID number In index file, size of key field is made fixed length Obviously, size of offset if also fixed length Therefore, entire key offset record is fixed length Can also store other data in index file (e.g., record length) 28

29 Music Index and Data Files key offset offset record ANG COL COL DGI DGI Index File 0 Lon 2312 Romeo & Juliet 62 RCA 2626 Quartet in C# 117 WAR 2369 Touchstone 1532 ANG 3795 Symphony #9 196 COL Nebraska Data File 29

30 Index Management With indexing, data files changes require index updates Addition. New records inserted into hole or appended Add new key offset pair to index Maintain index (in main memory) in key sorted order (for rapid search) Deletion. Use any previously discussed method to delete Remove record s key offset pair from index Update. Record s key changes, or it does not Former case, index s key value pair must be updated, repositioned Latter case, nothing changes in index 30

31 Secondary Key Index Secondary keys partition data file into groups Secondary key offsets are normally into primary key index Buffers secondary index Minimizes number of entries to update on data file modification Secondary key reference is a primary key To retrieve records, retrieve all primary key entries for secondary key, use primary key index to retrieve records Secondary index sorted by secondary key, within this by primary key 31

32 Music File Secondary Index key Beethoven Beethoven Beethoven Beethoven Corea Dvorak offset ANG3795 DGI39201 DGI8807 RCA2626 WAR23699 COL

33 Secondary Index Management Addition. Identical to primary key index addition Add new entry in sorted order to (each) secondary index Deletion. (Each) secondary index removes its reference to the deleted record Close holes in secondary index Even in main memory, can be expensive Alternatively, simply remove entry in primary key, stop Now, requests for deleted record through secondary index will fail on search of primary index With this approach, at some point secondary index should be cleaned up 33

34 Secondary Index Update Update. Secondary index references primary index, so some buffering is provided Three types of updates 1. Secondary key value updated Secondary index must update its corresponding entry, re-sort 2. Primary key value updated Search secondary index to update old primary key to new one, re-sort 3. Neither primary nor secondary key values updated No changes to secondary index required 34

35 Secondary Index Boolean Logic Secondary index can be used to perform Boolean logic searches Retrieve hit lists from searches on different secondary indices Merge results using Boolean logic operators E.g., find all recordings of Beethoven s Symphony #9 using secondary key indices for composer and recording Beethoven Symphony #9 Result ANG3795 ANG3795 ANG3795 DGI39201 COL31890 DGI8807 DGI8807 DGI8807 RCA

36 Secondary Index Improvements Various issues related to secondary key indices stored as simple arrays 1. Array must be reordered for each addition 2. Secondary key value is duplicated in many cases Different types of data structures can address these issues Inverted list, linked list 36

37 Inverted List Secondary index as secondary key, n primary keys If new record with same secondary key added, append record s primary key (in sorted order) Advantages Only need small local reorganization on new record addition Secondary keys are not duplicated Disadvantage Choosing size of n 37

38 Secondary Index Inverted List Key Beethoven Prokofiev References ANG3795 DGI39201 DGI8807 RCA2626 ANG LON2312 Secondary Key File 38

39 Linked List Use inverted list, but switch primary key array to linked list Don t create separate linked lists for each secondary key Build single primary key file referenced by secondary key index Each secondary key holds offset to first primary key in secondary key reference file First primary key holds offset to next primary key in reference list Similar to availability list for deleted records in data file 39

40 Secondary Index Linked List Key Offset Beethoven 30 Corea 70 Dvorak 90 Secondary Key File Offset Primary Key Next 0 LON RCA WAR ANG COL DGI MER COL DGI Primary Key Reference File 40

41 Linked List Performance Advantages Only updated secondary key index when record added or key updated Primary key reference list is entry sequenced, not sorted Primary key reference list uses fixed-length records, easy to implement deletion, direct access to location any record Disadvantage If primary key reference list stored on-disk, each key retrieved requires disk seek 41