Architecture of a Database Management System Ray Lockwood

Transcription

1 Assorted Topics Architecture of a Database Management System Pg 1 Architecture of a Database Management System Ray Lockwood Points: A DBMS is divided into modules or layers that isolate functionality. The is at the lowest level and interacts with the disk. The Buffer Manager translates disk blocks into pages containing a table s rows. The implement tree, heap, hash and index structures. The (or Query Processor) converts a SQL query into a runnable execution plan. How Data Is Stored On A disk drive is controlled by a processor that is either part of the disk drive itself or a separate disk controller card. This processor and its firmware are called the disk drive controller. The disk drive controller divides the physical disk into blocks. The controller maps the physical location of each block (expressed in hardware as a cylinder-head-sector number) into a sequential logical block number (0001, 0002, 0003,...). The controller presents the disk to the operating system as if it were a sequence of blocks. The operating system is composed of many modules, one of which is the file system. The file system organizes disk storage into files. A file is a sequence of bytes that resides somewhere on the disk. Cylinder-Head- Sector provides a threedimensional coordinate for a block in a stack of disk platters. Any individual byte in the file has an address. The byte s address is the starting address of the file, plus an offset, or number of bytes away from the file s starting point.

2 Assorted Topics Architecture of a Database Management System Pg 2 How Simple Database Products Store Tables Simple database products (desktop PC products) store their tables directly in a file under control of the file system. If a table s rows are 100 bytes long, and the database calls for, say, the 5 th row, the file system counts its way along the file to find the row: Start at the beginning of the file (offset 0). Count along the file until it comes to the beginning of the 5 th row (record 4) at an offset of 400 bytes from the starting address (read past the first 4 records at 100 bytes per record). Return the following 100 bytes. Relying solely on the file system for disk management hinders crash recovery and reduces performance. Offset 0 bytes Record 0 Offset 100 bytes Record 1 Offset 200 bytes Record 2 Offset 300 bytes Record 3 Offset 400 bytes Record 4 A simple DBMS system lays out records linearly on the disk. Start of file These records are 100 bytes long This method is effective and reliable, but it s not the fastest way to access rows in a table. How High Performance DMBS s Store Tables The type of DBMS we ve been studying is organized into layers as illustrated in the diagram at the right. The disk space manager stores data either in pre-allocated space in a file, or bypasses the file system altogether and uses raw disk space directly. Storage space is divided into pages, each of which is the same size as a disk block. The layers above the disk space manager see storage in terms of pages and never see the hardware or the filesystem. The disk space manager (not the operating system) decides the order that pages are written to and read from disk, and whether blocks are allocated contiguously. The disk space manager also keeps track of free blocks on the disk, and the mapping of pages to disk blocks. In short, the disk space manager takes over management of disk space from the file system and hides the underlying operating system and hardware from the rest of the DBMS. Many disk space managers don t rely on the file system at all but either do their own raw disk management, or extend the filesystem to provide required functionality. The Space Manager abstracts away the storage subsystem from the rest of the DBMS. Bypassing the file system makes the DBMS more portable between different OS/Hardware combinations

3 Assorted Topics Architecture of a Database Management System Pg 3 Buffer Manager The buffer manager maintains the buffer pool, the area of memory where working pages are kept. The database can be much larger than will fit in memory. When the DBMS needs a page that s not in the buffer, the buffer manager brings the page from disk and places it in a frame in the buffer pool. Users of large data banks must be protected from having to know how the data is organized in the machine -E. F. Codd Ideally, the entire database should fit into the buffer pool. Database servers should have lots of memory. If the buffer pool is already full, the buffer manager will decide which pages to replace acording to the replacement policy. Higher layers in the DBMS don t need to worry about whether the pages they need are in memory, the buffer manger will take care of fetching them. Buffer Manager Replaces OS Virtual Memory Functionality A DBMS uses a buffer manager instead of relying on the OS s virtual memory to handle swaping disk pages into and out of memory. The buffer manager can be much more efficient: The buffer manager predicts which pages will be needed next and pre-fetch them. The buffer manager implements more intelligent buffer replacement policies. The buffer manager forces pages to be written to disk to enhance durability. writing is done in a specific order and logged to enhance crash recovery. Examples of replacement policies are: Least Recently Used Most Recently Used FIFO Random Enormous work has gone into making the DBMS both fast and reliable.

4 Assorted Topics Architecture of a Database Management System Pg 4 Buffer Manager Housekeeping The buffer manager has to know two things about the pages in the buffer pool: Whether a page is still being used and can't be removed from the buffer pool. Whether a page has been changed and needs to be written to disk. To do this, the buffer manager keeps two variables for each page in the pool: Pin-count Dirty flag When a page in a frame is requested, the frame s pin-count is incremented. (The frame is pinned down by a user). When the page is released, the pin-count is decremented. If a frame has a zero pin-count, then the page can be released from the buffer if the space is needed by another page. When a page is modified and needs to be written to the disk, the dirty flag is set. If the page s space is needed by another page, the page is written to disk before it s released. Buffer Manager Crash Recovery The buffer manager has many mechanisms assure that minimal (if any) data loss will occur if the server crashes at any point during buffer processing. Techniques used include: Write-ahead log (part of transaction logging, transactions are written to disk before the page itself is written) Compensation log (logs rollbacks) Update log (contains before-images and after-images of modified pages for rollback and roll forward recovery) Transaction table (table of active transactions) Dirty page table (table of pages waiting to be written to disk) Pin Count: Tells whether the frame is in use or if it can be released from memory. Dirty Flag: Tells whether a frame has been modified and needs to be written to disk. The ACID property of Durability is taken very seriously. File and Access Methods The methods that implement the data structures described in an earlier lesson reside in this layer: Heaps Trees Hashes Indexes These methods operate on the pages in the buffer and have no contact with the hardware or the operating system. Even though these methods deal with data structures, they are abstracted away from the hardware.

5 Assorted Topics Architecture of a Database Management System Pg 5 This is a quick overview. The diagram at right expands the box. The query engine s job is to take a logical SQL query and produce a physical execution plan. Query Parser Query Optimizer Query Parser The query parser checks the SQL statement for valid syntax. (SQL error messages come from the query parser). Then it converts the SQL query into an intermediate language that the query optimizer can accept. Some parsers use relational calculus as the intermediate language. Plan Generator The plan generator converts the parsed query into a relational algebra query tree. Plan Cost Estimator Plan Generator Query Plan Evaluator π name Catalog Manager Relational calculus is a natural intermediate language for the query processor. SELECT Pilot.name FROM Pilot JOIN Airplane ON Pilot.ID = Airplane.PilotID WHERE Pilot.rating = 'VFR' AND Airplane.type = 1 rating='vfr' Λ type = 1 ID = PilotID Relational algebra in a query tree is an execution plan. The tree structure specifies the sequence which the relational algebra executes. The query tree is an execution plan. Execution starts at the bottom of the tree and moves upward. A large number of possible execution plans are derived from a single query. The plan generator produces many alternative plans and passes them to the plan cost estimator. Plan Cost Estimator Algebraic Query Tree The plan cost estimator evaluates the resource cost of each plan and chooses the one with the lowest cost. Depending on the environment, the plan cost estimator can choose a plan that requires, for example, the least memory usage, or the least execution time. Pilot Airplane Because many alternative execution plans are considered, the exact way you write a query probably won t affect the final execution. The selected query plan is likely to be very efficient. Query Plan Evaluator The query plan evaluator executes the chosen execution plan, and query processing is completed.