Lecture Data layout on disk. How to store relations (tables) in disk blocks. By Marina Barsky Winter 2016, University of Toronto

Transcription

1 Lecture Data layout on disk How to store relations (tables) in disk blocks By Marina Barsky Winter 2016, University of Toronto

2 How do we map tables to disk blocks Relation, or table: schema + collection of records SID: int Name: char (8) Age: int 11 Amy James 33 Record, or tuple: collection of fields 33 Lee 44 File: collection of disk blocks 8 KB block 8 KB block Block: collection of bytes (often in multiples of 4 or 8) 8 KB block 8 KB block

3 General representation of a single record (fixed-length) Schema address Schema is stored in System Catalogue Student (SID: INT, Name: CHAR (8), Age: INT) Record length Field offsets and the interpretation of bytes are according to the schema 16 Time Stamp 4 bytes 8 bytes 4 bytes Record Header Often, fields must be aligned at multiples of an address (4 bytes for 32- bit and 8 for 64-bit architectures)

4 General representation of a single record (fixed-length) Schema address Schema is stored in System Catalogue Student (SID: INT, Name: CHAR (8), Age: INT) Record length Field offsets and the interpretation of bytes are according to the schema 16 Time Stamp 4 bytes 8 bytes 4 bytes Record Header Just remember that each record has a header, we will not show it on the next slides

5 Multiple records inside the block (record header not shown) SID: int Name: char (8) Age: int 11 Amy James Lee 44 Slot 1 11 A m y Ɵ J a m e s Ɵ L e e Ɵ 44

6 Records inside the block: RID SID: int Name: char (8) Age: int 11 Amy James Lee 44 Block X 11 A m y Ɵ J a m e s Ɵ L e e Ɵ 44 Each record has unique record ID: RID = <block ID, slot #> Tuple (11, Amy, 22) RID = <X, 0> The exact starting position of each record Tuple (22, James, 33) RID = <X, 1> will be calculated by multiplying total Tuple (33, Lee, 44) RID = <X, 2> record length (16) by slot #

7 Records inside the block: RID cannot be changed SID: int Name: char (8) Age: int 11 Amy James Lee A m y Ɵ J a m e s Ɵ L e e Ɵ 44 Block X Once RID is assigned it cannot be changed because the record can be referenced from another place for example from an index

8 Deleting record <X, 1>: repacking SID: int Name: char (8) Age: int 11 Amy James Lee 44 Block X 11 A m y Ɵ L e e Ɵ 44 Tuple (33, Lee, 44) RID = <X, 2> now has a new RID: <X,1> This is unacceptable!

9 Adding block header: total occupied slots The block header starts from the end of the block, because it may grow, and the data slots may grow If the header would start at the beginning, each time we increase the header, we would need to move all the data slots and update their offsets 11 A m y Ɵ J a m e s Ɵ L e e Ɵ 44 Block X 3 Total occupied slots: can compute the beginning of a free space

10 Adding block header: bit-on to indicate free slot 11 A m y Ɵ J a m e s Ɵ L e e Ɵ 44 Block X Bitmap directory to indicate free and occupied slots

11 Deleting record <X, 1>: no repacking SID: int Name: char (8) Age: int 11 Amy James Lee A m y Ɵ J a m e s Ɵ L e e Ɵ 44 Block X Bit on to indicate that slot 1 is now free and can be reused

12 Inserting new record into a free slot SID: int Name: char (8) Age: int 11 Amy Lee Mary A m y Ɵ M a r y Ɵ L e e Ɵ 44 Block X

13 Variable-length records: motivation Student (SID: INT, Name: VARCHAR (255), Age: INT) Slotted block (page) organization works only for fixed-length records: only when we know the total length of each record (R) we can treat each cluster of R bytes as a slot Consider changing the size of the Name attribute, expecting longer names It would be wasteful to use 255 bytes for each name even when some of them are only 3 bytes long!

14 Storing a variable-length record: offset directory in record header Schema address Schema is stored in System Catalogue Student (SID: INT, Name: VARCHAR (255), Age: INT) Record length 4? 4 Field offsets 20 Time Stamp bytes 12 bytes 4 bytes Record Header

15 Storing a variable-length record: offset directory in record header Schema address Schema is stored in System Catalogue Student (SID: INT, Name: VARCHAR (255), Age: INT) Record length 4? 4 20 Time Stamp Record Header bytes 12 bytes 4 bytes Offset into the beginning of the next field: from the beginning of the data

16 Storing multiple variable-length records: with field offsets SID: int Name: char (8) Age: int 11 Amy Jonathan Lee 44 Total: A m y Ɵ 22 Total: J o n a t h a n Ɵ 55 Total: L e e Ɵ 44

17 Storing multiple variable-length records: specify start of each record in block header SID: int Name: char (8) Age: int 11 Amy Jonathan Lee 44 Total: A m y Ɵ 22 Total: J o n a t h a n Ɵ 55 Total: L e e Ɵ Start of free space

18 Storing multiple variable-length records: specify start of each record in block header SID: int Name: char (8) Age: int 11 Amy Jonathan Lee 44 Total: A m y Ɵ 22 Total: J o n a t h a n Ɵ 55 Total: L e e Ɵ Total records in block

19 Storing multiple variable-length records: specify start of each record in block header SID: int Name: char (8) Age: int 11 Amy Jonathan Lee 44 Slot 1 Total: A m y Ɵ 22 Total: J o n a t h a n Ɵ 55 Total: L e e Ɵ Slot Starts for each record slot

20 Slot directory preserves RID SID: int Name: char (8) Age: int 11 Amy Jonathan Lee 44 Total: A m y Ɵ 22 Total: J o n a t h a n Ɵ 55 Total: L e e Ɵ Slot Now if a record has changed the physical location on the page, the only thing that changes is its entry in the slot directory

21 Variable-length records: deletion SID: int Name: char (8) Age: int 11 Amy Jonathan Lee 44 Total: Total: J o n a t h a n Ɵ L e e Ɵ Slot The record is deleted, the remaining records are repacked, and the header updated. Note that tuple (33, Lee, 44) still has RID = <X, 2>

22 Variable-length records: deletion SID: int Name: char (8) Age: int 11 Amy Jonathan Lee 44 Total: Total: J o n a t h a n Ɵ L e e Ɵ 44 Slot 0 can be reused for another record Slot

23 Slot directory: efficient representation of nulls SID: int Name: char (8) Age: int 44 Jonathan NULL 44 Total: Total: J o n a t h a n Ɵ Slot To indicate that field has value NULL (unspecified, non-applicable, not set), we just set 2 offsets to the same value

24 Storing records by row and by column NSM: N-ary Storage Model i.e. the slotted page model DSM: Decomposition Storage Model DSM better space utilization and works faster for smaller projections

25 Combining pages (blocks) into files Which block of a file should a record go to? I. Anywhere? Heap organization II. How to search for SID= 123? Sorted by some key? Sequential organization Keeping it sorted could be painful III. Based on a hash key? Hashing organization Store the record with SID = x in the block number h(x)%1000

26 I. Heap files Heap files unordered set of disk pages simplest file structure. Contains records in no particular order. As file grows and shrinks, disk pages are allocated and de-allocated. To support record level operations, we must: Keep track of the pages in a file Keep track of free space on pages Keep track of the records on a page (we know how to do that)

27 Heap file implemented as doublylinked list Data Page Data Page Data Page Full Pages Header Page Data Page Data Page Data Page Pages with Free Space The header page id and Heap file name must be stored someplace. The header page contains 2 pointers block IDs Each page contains 2 `pointers in addition to page header and data as described before.

28 Heap file with page directory The entry for a page can include the number of free bytes on the page. The directory is a collection of pages by itself : linked list implementation is just one alternative. Much smaller than linked list of all HF pages! Header Page DIRECTORY Data Page 1 Data Page 2 Data Page N

29 II. Sequential (sorted) file organization Keep sorted by some search key Insertion Find the block in which the tuple should be If there is free space, insert it Otherwise, create an overflow page, and link from the corresponding page. Deletions Can create a long list of overflow pages Delete and keep the record of a free space Databases tend to be insert-heavy, so free space gets used fast Can become fragmented Must reorganize once in a while

30 III. Hash-based file organization Allocate file with 4 pages Store record with search key k in block number h(k) % 4, where h(k) is a hash function (1000, A, ) (200, B, ) (4044, C, ) (401, Ax, ) (21, Bx, ) Block 0 Block 1 Buckets Blocks are called buckets What if the bucket becomes full? Overflow pages. As file grows, the search becomes inefficient (1002, Ay, ) (10, By, ) (1003, Az, ) (35, Bz, ) Block 2 Block 3

31 In practice: example of a Heap file implementation 0 1 Header: Total data blocks: 1 Next free block: 1 In a separate block encode file header information: Total N of blocks Initially: a relative address of the first data page Write to disk the first data block (without actual data)

32 Example of a Heap file implementation: double-linked list Total blocks: 2 To full block: 1 To free block: 2 Data Prev: 0 Next: / Prev: 0 Next: / The new data is written to the first free block Suppose that it becomes full Reference to it becomes a link to a full block A new free block is appended to the file

33 Example of a Heap file implementation: double-linked list Total blocks: 2 To full block: 1 To free block: 4 Data Prev: 0 Next: 2 Prev: 1 Prev: 2 Prev: 0 Data Data Next:3 Next:/ Next:/ Suppose we only insert new data The file is growing and the links between pages are updated

34 Example of a Heap file implementation: double-linked list Total blocks: 2 To full block: 1 To free block: 2 Data Prev: 0 Next: 3 Prev: 0 Prev: 1 Prev: 2 Data Data Next:4 Next:/ Next:/ Now we need to delete some data from block 2 We just mark its slot as free, and we update the links to free blocks and full blocks accordingly Next time we need to insert data we reuse this slot

35 How to override a block with new data (C example) /* Supposing that you formatted the file already and the file is opened. */ void modifyscore ( FILE * fp, int SID, int score ) { StudentRecord * blockbuffer = ; //allocate a buffer of size 1 block //determine a block ID and slot number (from an index or from a scan) int blockid =...; int slotnumber = //Seek the file for block blockid and read the block into buffer fseek ( fp, blockid * sizeof (blockbuffer), SEEK_SET ); fread (blockbuffer, 1, sizeof (blockbuffer), fp); //find the student inside the block by slotid and update its score field student.score = score; //Seek back because the pointer moved during read. fseek( fp, (-1)* sizeof (blockbuffer ), SEEK_CUR ); //Overwrite his information by writing back the block fwrite (blockbuffer, sizeof(blockbuffer ), 1, fp ); typedef struct{ int SID; char name[32]; int score } StudentRecord;

36 File manager component of DBMS The file manager component takes care of file manipulations It interacts directly with the disk blocks, bypassing operating system It is a complex piece of software whose detailed implementation is outside the scope of this course However, you can implement file storage optimizations, developing a simulated example in the previous slide

37 Comparing efficiency of file organizations Operations to compare: Scan: fetch all records from disk Equality search: find record with key = k Range selection: find all records where key is between a and b Insert a record Delete a record

38 Cost Model for Our Analysis N total number of data pages (file blocks) We calculate the average number of disk I/Os per operation We ignore CPU costs in our model Measuring number of page I/O s ignores differences between random and sequential I/Os Average-case analysis; based on several simplistic assumptions. Good enough to show the overall trends!

39 Assumptions Heap Files: Equality selection on key - exactly one match. Sorted Files: Files compacted after deletions, no overflow pages Hash: No overflow buckets.

40 Cost of operations for different file organizations (in disk I/Os) N - number of data pages (1) Heap (2) Sorted (3) Hashed Scan Equality Range Insert Delete

41 Cost of operations for different file organization (in disk I/Os) N - number of data pages Scan Equality Range Insert Delete (1) Heap N 0.5N N 2 0.5N +1 (2) Sorted N log 2 N log 2 N + output log 2 N + N (3) Hashed N 1 N 2 2 log 2 N + N * Several assumptions underlie these (rough) estimates!

42 Cost of operations for different file organization (in disk I/Os) N - number of data pages Scan Equality Range Insert Delete (1) Heap N 0.5N N 2 0.5N +1 (2) Sorted N log 2 N log 2 N + output log 2 N + N (3) Hashed N 1 N 2 2 log 2 N + N Always insert in the end of the file: 1 I/O to read, 1 to write back

43 Cost of operations for different file organization (in disk I/Os) N - number of data pages Scan Equality Range Insert Delete (1) Heap N 0.5N N 2 0.5N +1 (2) Sorted N log 2 N log 2 N + output log 2 N + N (3) Hashed N 1 N 2 2 log 2 N + N Find the page (average equality search, mark record as deleted and write back

44 Cost of operations for different file organization (in disk I/Os) N - number of data pages Scan Equality Range Insert Delete (1) Heap N 0.5N N 2 0.5N +1 (2) Sorted N log 2 N log 2 N + output log 2 N + N (3) Hashed N 1 N 2 2 log 2 N + N Find the page insert record, shift all the pages, assuming there are no empty slots

45 Cost of operations for different file organization (in disk I/Os) N - number of data pages Scan Equality Range Insert Delete (1) Heap N 0.5N N 2 0.5N +1 (2) Sorted N log 2 N log 2 N + output log 2 N + N (3) Hashed N 1 N 2 2 log 2 N + N Read and write back

46 All these file organizations are not very efficient What if we want to find a particular record by value? Account info for SIN = 123 Binary search in sequential file Takes log(n) disk accesses Random accesses Too much For N = 1,000,000,000 log(n) = 30 Recall each random access 10 ms 300 ms to find just one account information! < 4 requests satisfied per second

47 Index A data structure for efficient search through large databases. Think - library index/catalogue Two key ideas: The records are mapped to the disk blocks in specific ways Auxiliary data structures are maintained that allow quick search Search key - attribute or set of attributes used to look up records - e.g. SID for a Students table Two types of indexes Ordered indexes Hash-based indexes

48 DBMS vs. OS File System OS does disk space & buffer mgmt: why not let OS manage these tasks? Differences in OS support: portability issues Ability to dynamically update specific pages, to keep track of pages with free space, to control order of pages Need an access to a particular block for example in hashing organization Each file page in DBMS contains an additional information and requires a special management OS files can t span multiple disks. Buffering DBMS files is quite different (see Buffering lecture)

49 Summary Variable-length record format with field offset directory offers support for direct access to i th field and efficient representation of null values. Slotted page format supports variable-length records and allows records to move on page without changing its RID. Heap, sequential, and hash-based file organizations are not very efficient in many cases We need more sophisticated file organizations and auxiliary data structures

50 Handling large amount of data efficiently 1. Storage media and its constraints (magnetic disks, buffering) 2. Algorithms for large inputs (sorting) 3. Data structures (trees, hashes and bitmaps)