Storage and Indexing, Part I

Transcription

1 Storage and Indexing, Part I Computer Science E-66 Harvard University David G. Sullivan, Ph.D. Accessing the Disk Data is arranged on disk in units called blocks. typically fairly large (e.g., 4K or 8K) Relatively speaking, disk I/O is very expensive. in the time it takes to read a single disk block, the processor could be executing millions of instructions! The DBMS tries to minimize the number of disk accesses.

2 Review: DBMS Architecture A DBMS can be viewed as a composition of two layers. At the bottom is the storage layer or storage engine, which takes care of storing and retrieving the data. logical layer storage engine Above that is the logical layer, which provides an abstract representation of the data. OS disks FS Logical-to-Physical Mapping The logical layer implements a mapping between: the logical schema of a database its physical representation logical layer storage engine OS FS In the relational model, the schema includes: attributes/columns, including their types tuples/rows relations/tables disks To be model-neutral, we'll use these terms instead: field for an individual data value record for a group of fields collection for a group of records

3 Logical-to-Physical Mapping (cont.) A DBMS may use the filesystem, or it may bypass it and use its own disk manager. logical layer storage engine OS FS In either case, a DBMS may use units called pages that have a different size than the block size. can be helpful in performance tuning disks Logical-to-Physical Mapping (cont.) Need to consider several different issues: how to map logical records to their physical representation how to organize records on a page how to organize collections of pages called files, which may or may not correspond to OS files including the use of index structures what metadata is needed and where should it be stored? example: the types and lengths of the fields may need both per-record and per-collection metadata

4 Fixed- or Variable-Length Records? This choice depends on: the types of fields that the records contain the number of fields per record, and whether it can vary Simple case: use fixed-length records when all fields are fixed-length (e.g., CHAR or INTEGER), there is a fixed number of fields per record Fixed- or Variable-Length Records? (cont.) The choice is less straightforward when you have either: variable-length fields (e.g., VARCHAR) a variable number of fields per record (e.g., in XML) Two options: 1. fixed-length records: always allocate comp sci the maximum possible length math plusses and minuses: + less metadata is needed, because: every record has the same length a given field is in a consistent position within all records + changing a field's value doesn't change the record's length thus, changes never necessitate moving the record we waste space when a record has fields shorter than their max length, or is missing fields

5 Fixed- or Variable-Length Records? (cont.) 2. variable-length records: only allocate the space that each record actually needs plusses and minuses: comp sci math more metadata is needed in order to: determine the boundaries between records determine the locations of the fields in a given record changing a field's value can change the record's length thus, we may need to move the record + we don't waste space when a record has fields shorter than their max length, or is missing fields Format of Fixed-Length Records With fixed-length records, the fields can be stored consecutively. If a fixed-length record contains a variable-length field, we allocate the max. length and store the actual length. field 1 field 2 field comp sci# math# comp sci math 125 O 2 O 3 OR To find the position of a field, use per-collection metadata. typically store the offset of each field how many bytes the field is from the start of the record (see O 2 and O 3 above)

6 Format of Variable-Length Records With variable-length records, we need per-record metadata to determine the locations of the fields. For simplicity, we ll assume all records in a given collection have the same # of fields. We'll look at how the following record would be stored: CHAR(7) VARCHAR(20) INT (' ', 'comp sci', 200) assumptions: each character in a string takes up 1 byte each integer takes up 4 bytes, including both: the integers in the data (e.g., the 200 above) any integer metadata offsets, field lengths, etc. Format of Variable-Length Records (cont.) We'll consider three types of operations: 1. finding/extracting the value of a single field SELECT dob FROM Person WHERE name = 'Julianne Moore'; 2. updating the value of a single field its length may become smaller or larger 3. adding a new field to end of each record (e.g., adding a new column to a table)

7 Format of Variable-Length Records (cont.) Option 1: Terminate field values with a special delimiter character. CHAR(7) VARCHAR(20) INT # comp sci # 200 # 1. finding/extracting the value of a single field this is very inefficient; need to scan byte-by-byte to: find the start of the field we re looking for determine the length of its value (if it is variable-length) 2. updating the value of a single field if it changes in size, we need to shift the values after it, but we don't need to change their metadata 3. adding a new field to the end of each record no need to change the existing field values or metadata Format of Variable-Length Records (cont.) Option 2: Precede each field by its length. CHAR(7) VARCHAR(20) INT comp sci finding/extracting the value of a single field this is more efficient can jump over fields, rather than scanning byte-by-byte (but may need to perform multiple jumps) never need to scan to determine the length of a value 2. updating the value of a single field same as option 1 3. adding a new field to the end of each record same as option 1

8 Format of Variable-Length Records (cont.) Option 3: Put offsets and other metadata in a record header. # of bytes from the start of the record comp sci 200 record header computing the offsets 3 fields in record 4 offsets, each of which is a 4-byte int thus, the offsets take up 4*4 = 16 bytes offset[0] = 16, because field[0] comes right after the header offset[1] = 16 + len(' ') = = 23 offset[2] = 23 + len('comp sci') = = 31 offset[3] = offset of the end of the record = (since 200 an int) = 35 We store this offset because it may be needed to compute the length of a field's value! Format of Variable-Length Records (cont.) Option 3 (cont.) comp sci finding/extracting the value of a single field this representation is the most efficient. it allows us to: jump directly to the field we're interested in compute its length without scanning through its value 2. updating the value of a single field less efficient than options 1 and 2 if the field's length changes. need to shift subsequent values and recompute their offsets. 3. adding a new field to the end of each record need to recompute all offsets in all records. why?

9 Format of Variable-Length Records (cont.) Option 3 (cont.) comp sci finding/extracting the value of a single field this representation is the most efficient. it allows us to: jump directly to the field we're interested in compute its length without scanning through its value 2. updating the value of a single field less efficient than options 1 and 2 if the field's length changes. need to shift subsequent values and recompute their offsets. 3. adding a new field to the end of each record need to recompute all offsets in all records. why? adding a new field makes the header longer and thus moves all fields! we can avoid this by: putting the field offsets at the end of the record putting an offset to the field offsets at the start comp sci Representing Null Values Option 1: add an "out-of-band" value for every data type con: need to increase the size of most data types, or reduce the range of possible values Option 2: use per-record metadata 2a: for some fields, a length of 0 could indicate NULL when wouldn't this work? ( 2b: add additional metadata to the record to track nulls con: waste spaces in collections with few null values 2c: if using an array of offsets, use a special offset (e.g., 0 or -1) comp sci

10 Representing Null Values Option 1: add an "out-of-band" value for every data type con: need to increase the size of most data types, or reduce the range of possible values Option 2: use per-record metadata 2a: for some fields, a length of 0 could indicate NULL when wouldn't this work? ( strings (need to allow for empty strings); fixed-length fields for which we don't store the length 2b: add additional metadata to the record to track nulls con: waste spaces in collections with few null values 2c: if using an array of offsets, use a special offset (e.g., 0 or -1) comp sci Practice: Which is the correct record header? We're inserting the following row into a simplified Movie table: CHAR(7) VARCHAR(64) INT VARCHAR(5) INT (' ', 'Moonlight', 111, 'R', NULL) and we're using: -1 for NULL the same assumptions about data types as before A. B. C. D E. none of these

11 Which is the correct record header? We're inserting the following row into a simplified Movie table: CHAR(7) VARCHAR(64) INT VARCHAR(5) INT (' ', 'Moonlight', 111, 'R', NULL) and we're using: -1 for NULL the same assumptions about data types as before A. B columns in table 6 offsets the offsets take up 6*4 = 24 bytes C. D E. none of these Which is the correct record header? We're inserting the following row into a simplified Movie table: CHAR(7) VARCHAR(64) INT VARCHAR(5) INT (' ', 'Moonlight', 111, 'R', NULL) and we're using: -1 for NULL the same assumptions about data types as before A. B. C columns in table 6 offsets the offsets take up 6*4 = 24 bytes thus, offset[0] = 24 D E. none of these

12 Which is the correct record header? We're inserting the following row into a simplified Movie table: CHAR(7) VARCHAR(64) INT VARCHAR(5) INT (' ', 'Moonlight', 111, 'R', NULL) and we're using: -1 for NULL the same assumptions about data types as before A. B. C Moonlight columns in table 6 offsets the offsets take up 6*4 = 24 bytes thus, offset[0] = 24 offset[1] = 24 + len(' ') = 31 D E. none of these Which is the correct record header? We're inserting the following row into a simplified Movie table: CHAR(7) VARCHAR(64) INT VARCHAR(5) INT (' ', 'Moonlight', 111, 'R', NULL) and we're using: -1 for NULL the same assumptions about data types as before A. B. C. D Moonlight E. none of these 5 columns in table 6 offsets the offsets take up 6*4 = 24 bytes thus, offset[0] = 24 offset[1] = 24 + len(' ') = 31 offset[2] = 31 + len('moonlight') = 40

13 Which is the correct record header? We're inserting the following row into a simplified Movie table: CHAR(7) VARCHAR(64) INT VARCHAR(5) INT (' ', 'Moonlight', 111, 'R', NULL) and we're using: -1 for NULL the same assumptions about data types as before A. B. C. D Moonlight 111 R E. none of these 5 columns in table 6 offsets the offsets take up 6*4 = 24 bytes thus, offset[0] = 24 offset[1] = 24 + len(' ') = 31 offset[2] = 31 + len('moonlight') = 40 offset[3] = (for an int) = 44 Which is the correct record header? We're inserting the following row into a simplified Movie table: CHAR(7) VARCHAR(64) INT VARCHAR(5) INT (' ', 'Moonlight', 111, 'R', NULL) and we're using: -1 for NULL the same assumptions about data types as before A. B. C. D Moonlight 111 R E. none of these 5 columns in table 6 offsets the offsets take up 6*4 = 24 bytes thus, offset[0] = 24 offset[1] = 24 + len(' ') = 31 offset[2] = 31 + len('moonlight') = 40 offset[3] = (for an int) = 44 offset[4] = -1 for NULL

14 Which is the correct record header? We're inserting the following row into a simplified Movie table: CHAR(7) VARCHAR(64) INT VARCHAR(5) INT (' ', 'Moonlight', 111, 'R', NULL) and we're using: -1 for NULL the same assumptions about data types as before A. B. C. D Moonlight 111 R E. none of these 5 columns in table 6 offsets the offsets take up 6*4 = 24 bytes thus, offset[0] = 24 offset[1] = 24 + len(' ') = 31 offset[2] = 31 + len('moonlight') = 40 offset[3] = (for an int) = 44 offset[4] = -1 for NULL offset[5] = end of record = 44 + len('r') = 45 Page Format: How to Group Records Together Recall that records are grouped together on pages. Each record must have a record ID or rid that allows us to: find the page that the record is on find the location of the record on that page We will again consider different schemes for fixed-length and variable-length records.

15 Page Format with Fixed-Length Records Can use a fairly simple scheme: let b = the length of each record in bytes the maximum # of records per page is known as the fill factor, f = page_size / b ex: 4K pages, b = 512 bytes f = 4096/512 = 8 page 0 0: page 1 0: Use simple math to find any record. ex: find rid 13 page # = rid / f = 13 / 8 = 1 index on page, i = rid % f = 13 % 8 = 5 offset from start of page = i * b = 5 * 512 = 2560 bytes Can we move a record to a different location on the same page? 1: 2: 3: 4: 5: 6: 7: 1: 2: 3: 4: 5: rid 13 6: 7:

16 Page Format with Fixed-Length Records Can use a fairly simple scheme: let b = the length of each record in bytes the maximum # of records per page is known as the fill factor, f = page_size / b ex: 4K pages, b = 512 bytes f = 4096/512 = 8 page 0 0: page 1 0: Use simple math to find any record. ex: find rid 13 page # = rid / f = 13 / 8 = 1 index on page, i = rid % f = 13 % 8 = 5 offset from start of page = i * b = 5 * 512 = 2560 bytes Can we move a record to a different location on the same page? no. that changes the rid, which may be stored elsewhere 1: 2: 3: 4: 5: 6: 7: 1: 2: 3: 4: 5: rid 13 6: 7: Page Format with Fixed-Length Records (cont.) To deal with deletions and insertions: add a bit to each record indicating whether it's been deleted so that it won't be visited during scans of all records when inserting, use a free list: store the location of the first "empty" record (either never used or deleted) in a special header page each empty record points to the next one

17 Page Format with Variable-Length Records The fixed-length scheme breaks down because: we don t know how many records will fit on a page the offsets of records on a page will vary from page to page One possible approach: let rid = (page #, offset within page) problem: can t move a record without changing its rid when would we need to move it? when an update changes its size 0: 70 bytes 1: 80 bytes to make room for other records 0: 50 bytes 20 bytes 1: 60 bytes 20 bytes example: what if record 0 is updated and is now 100 bytes? example: record 0: bytes record 1: bytes we have 40 free bytes, but can t use them for records larger than 20 unless we move rec 1 Page Format with Variable-Length Records (cont.) A better approach: add a header an array of offsets to each page rid = (page #, index into offset array) when we move a record on the page, we change its offset but we don t change its rid! page 10: (10, 0) (10, 2) (10, 1) record (10, 1) grows in size (10, 0) (10, 1) (10, 2)

18 Page Format with Variable-Length Records (cont.) Problem: how large an offset array should we preallocate? Solution: don t preallocate! start adding records at the end of the page and work backwards toward the beginning doing so allows us to grow the offset array as needed page 10: 0 page 10: 0 1 (10, 0) (10, 0) (10, 1) page 10: (10, 0) (10, 2) (10, 1) Other Issues If a record becomes too large to fit on a page: move it to another page put its new location in the offset-array entry given by its rid (10, 0) (10, 1) (10, 2) (10, 2) grows page 10: page 35: , 0 (10, 0) (10, 1) (10, 2) If a record is too large to fit on any page, we can put it in one or more special pages called overflow pages. If a record is deleted, we put a special value in its offset-array entry so that scans through the collection will not try to access it. The page header needs to include info. that can be used to determine where the free space is on the page.

19 Database Files A logical collection of database pages is referred to as a file. may or may not correspond to an OS file The DBMS needs to manage the pages within a given file: keeping track of all used pages, to support scans through all records in the collection keeping track of unused pages and/or used pages with some free space, to support efficient insertions Index Structures An index structure stores (key, data) pairs. also known as a dictionary or map we will sometimes refer to the (key, data) pairs as items The index allows us to more efficiently access a given record. quickly find it based on a particular field instead of scanning through the entire collection to find it A given collection of records may have multiple index structures: one clustered or primary index some number of unclustered or secondary indices

20 Clustered/Primary Index The clustered index is the one that stores the full records. also known as a primary index, because it is typically based on the primary key If the records are stored outside of an index structure, the resulting file is sometimes called a heap file. managed somewhat like the heap memory region Unclustered/Secondary Indices In addition to the clustered/primary index, there can be one or more unclustered indices based on other fields. also known as secondary indices Example: Customer(id, name, street, city, state, zip) primary index: (key, data) = (id, all of the remaining fields) a secondary index to enable quick searches by name (key, data) = (name, id) does not include the other fields! We need two lookups when we start with the secondary index. example: looking for Ted Codd's zip code search for 'Ted Codd' in the secondary index '123456' (his id) search for '123456' in the primary index his full record, including his zip code