Overview of Project V Buffer Manager. Steps of Phase II

Transcription

1 Buffer Manager: Project V Assignment UC Berkeley Computer Science 186 Fall 2002 Introduction to Database Systems November 15, 2002 Due Tuesday, December 3, PM Overview of Project V Buffer Manager This project will add functionality to the backend code of PostgreSQL. Projects I and II showed you how to use a database, Project III extended the database through user code, and Project IV examined the optimizer and how the database decides how to process queries. Now we delve into the actual code. Due to time constraints, we limit our investigation to one core module, the buffer manager. The actual coding for this project is minimal. The major parts of this project are: Understanding different memory management strategies Examining and understanding existing code Due to disk space constraints, we are providing a leaner PostgreSQL source tree. Some features (contributed utilities, regression tests, manual, tutorials and other language support) have been removed. You will not needs these for this project, but you are free to download them if you have the space. Even with these modifications, you will not have extra space to store other files, so be sure to remove them before beginning. Make sure your code runs smoothly on EECS instructional machines (rhombus, pentagon, and torus.cs.berkeley.edu) prior to submission since ALL grading will done on those. However, since all the software is freely available, you may find it easier to develop your code/website at home or on other machines you have access to. The instructions provided are designed to work with the EECS instructional machines and may not work as smoothly elsewhere. The TAs and Professor will only support EECS instructional machines. Steps of Phase II 1. Learn different Strategies (20%) 2. Compiling and Testing PostgreSQL 3. Examine Code & Implement MRU (30%) 4. Implement 2Q Simplified (50%) 5. (Optional) Implement 2Q Full (10% Bonus) 6. Submit Please spend some time to read this assignment completely before beginning. 1

2 Learn Different Strategies (20%) The textbook describes LRU and MRU strategies in section These are two basic strategies that work well enough for most systems. However, some work has shown that second chance algorithms or those with more history can work better for some workloads. We will examine, 2Q. You should read the first four pages (up to Section 3) of the original research paper introducing 2Q. It can be downloaded from OR You are required to learn only the simplified version. Understanding the full version is optional. Hint: in 2Q, the queues are not fixed length. Now we will do a small exercise to test your knowledge of the algorithms. For LRU, MRU, and 2Q Simplified trace the buffer contents for the workload listed below. For your solutions, list each time that a memory access caused a page fault, and which (if any) page is thrown out of memory to make room. Record your answers in a file called strategies. Start with LRU and list the time, a tab, and the page (if any) removed from memory. Then do MRU and 2Q leaving a blank line in between. For example: Time T1 T2 T5 Page removed W Z LRU T2 T5 Z MRU T4 2Q Simplified T5 W Notice the first page faults did not remove a page, so that column is blank Assume there are 4 pages your buffer manager must manage. For 2Q, the tunable (threshold) parameter is 2. All four pages are empty to start. For each access the page is pinned, and then immediately unpinned 1. T1 A T7 B T13 A T19 B T2 B T8 E T14 B T20 E T3 C T9 F T15 C T21 A T4 D T10 A T16 G T22 C T5 A T11 G T17 G T23 D T6 A T12 B T18 F T24 A For 2Q Simplified there are 12 page faults leaving ACDE in the buffers at the end. 1 You can not make this same assumption when writing the actual code. A page may be pinned for any length of time 2

3 Compiling and Testing PostgreSQL You can unpack the source code for this project by running the following command: tar xf ~cs186/src/postgresql-small tar If you have the space, you can use the complete source code at tar xf ~cs186/src/postgresql tar Both archives will unpack to a directory called postgresql The trimmed version requires about 40MB to unpack and compile. The full version requires about 50MB. No code was changed between these versions and the actual version. The trimmed version just has some directories (features) removed and the Makefiles have been adjusted properly. To compile 1../configure [takes about 2 minutes] 2. gmake [takes about 5 minutes] After modifying the code, you only need to run gmake again. We will provide a basic test harness, which directly runs the buffer manager freelist code. This can partially test the correctness of the algorithms, but may not find other errors (such as breaking existing code). Your code is required to compile with the provided test harness AND run correctly when using the full PostgreSQL application, i.e. it can properly run queries. Be sure to test it thoroughly, including possible edge cases. The test harness code is available in ~cs186/buftest/*. Copy the code to src/buftest and run make to test your compile the tester. You may want to modify the tester to add additional test cases. The actual grading tester will be significantly different; the one provided is only for your convenience (and to make sure your code will compile with our tester)! To run your newly compiled version of PostgreSQL (without the tester), you can either adjust your paths so that our binary is not used (type which postgres to find out which one you are running). The other option is to run the backend server directly. To start the backend server, run src/backend/postgres s <dbname>. It will use the same data directories (and data) already present (and used by the pre-compiled server). The interface is directly to the backend, so psql features will not work. If your binary does not work correctly, it may corrupt the data, so be warned. Through the backend, you can directly type SQL statements, although the output is somewhat painful to read. To exit type CTRL-D. In addition, the s option will tell the server to print statistics at the end of each query, including the number of page faults! Finally, you can restrict the number of buffers PostgreSQL will use by adding the B <nbuffers> option so that even small queries generate page faults. More information on using the backend directly is available at 3

4 Examine Code & Implement MRU (30%) The actual implementation of MRU is rather straightforward once you understand the existing code. The existing code is not extremely clear, but it is not unreadable. It may take you a few hours (or more) to understand it. Since understanding the existing code is a significant part of the assignment, the TAs and Professor will not assist you in understanding the existing code (beyond what we discuss here). The actual buffer manager code is neatly separated from the rest of the codebase. Located in src/backend/storage/buffer and src/include/storage we are interested in modifying only two files. freelist.c which manages which pages are not pinned in memory, and which pages eligible for replacement. buf_internals.h which contains the definition for each buffer description (called BufferDesc). Most of the fields in BufferDesc are managed by other parts of the code, but for efficiency some fields are used to manage the buffer space while it is eligible for replacement (when it s on the existing LRU freelist). Some initialization of the queue occurs in buf_table.c. However, you must do all your initialization in freelist.c. A page is considered eligible for replacement when all its pins have been removed. It is also at that time that the page is time stamped and considered last accessed. You may modify these two files to change the implementation from the existing LRU queue to the MRU discipline. You may find that you do not need to edit both files or change many of the functions. Unfortunately there is no partial credit for this section. Be sure it is correct. A bad buffer manager makes the rest of the system useless! Implement 2Q Simplified (50%) Now that you are more acquainted with the code, you can implement 2Q Simplified as described in the pseudo code on Page 441 of the 2Q paper. The tunable (threshold) parameter should be set to ½ the total number of buffers (rounded down). We recommend returning to the original code, and starting from there. You may modify freelist.c and buf_internals.h to change the implementation to the LRU-2Q Simplified discipline. Again, you may find (depending on your implementation) that you do not need to edit both files. You may find that the original comments in the source code are not completely correct (or more precisely, too optimistic). They probably did not consider many other strategies when writing the comments. 4

5 There is limited partial credit available for code that does not work correctly. Horribly inefficient implementation may be docked some points. Partial credit is only available to those who recognize it is not working, and fully document the problem (and possible solutions) in the README file. Implement 2Q Full (10% Bonus) This portion is optional, and if completed, can help raise your project average if you did not do as well on other projects as you wanted. You do not need to complete this section to earn an A. Implement the 2Q Full strategy as listed on Page 442 of the 2Q paper. We recommend adjusting your existing 2Q Simplified code for this. (Be sure to save the working 2Q simplified code for your submission!) There is no partial credit for code that does not work or is horribly inefficient. Submit You must submit at least 6 files (even if they are incomplete or unchanged). They are 1. README what works, what doesn t, how many slip days, etc. 2. strategies answers to the first part, in the correct format! 3. freelist.c.mru MRU implementation 4. buf_internals.h.mru MRU implementation 5. freelist.c.s2q Simplified 2Q implementation 6. buf_internals.h.s2q Simplified 2Q implementation If you choose to submit the bonus section, you may include two additional files 1. freelist.c.f2q Full 2Q implementation 2. buf_internals.h.f2q Full 2Q implementation Use submit Project-V to submit the files. Be sure not to mix the files up. Additional files will not be accepted. Each group only needs to submit once. If there are multiple submissions, the last submission by any group member will be used for grading and the calculation of slip days. 5