Web Servers /Personal Workstations: Measuring Different Ways They Use the File System

Transcription

1 Web Servers /Personal Workstations: Measuring Different Ways They Use the File System Randy Appleton Northern Michigan University, Marquette MI, USA Abstract This research compares the different ways in which web servers and personal workstations use the file system. The differences between web server file usage and personal workstation file usage is described The dynamic distributions of file types opened while the systems run, dynamic distributions of file locations, and frequency of various system calls are all presented and compared between web servers and personal workstations. 1. Introduction In this research we analyze several Linux servers and workstations to determine their usage of various file system calls while running their normal daily workload. The file system calls used on actual servers were identified. The kernel was modified to measure parameters such as usage count, transfer size and error rate. Additional data was gathered about the files actually used on a dynamic basis such as file location, name and operation, yielding a file system trace. The results of this study are an analysis of actual file system call usage combined with a description of the types, names and locations of files used. The web servers in question process 20,000 hits per day, involving 5,000 unique hosts per week and 930 megabytes/day of data transfer. The personal workstations were the workstations of people who compile, edit papers, browse the web, and do other daily office work. This project was done in the hope that kernel designers can use the data within to better optimize file system design and algorithm selection. Further we hope that by showing web site designers and web application designers their actual file usage patterns they too can improve their code. Finally, we wanted to show system administrators the different loads they place on their machines. 2. Related Work Several studies have analyzed file system usage. File systems usage was captured and analyzed for Unix in 1984 in [Zou84], for BSD in 1985 in [Ous85], and Linux in 2004 in [Ara04]. The author has done previous research in this area in the context of file system prefetching in [Gri94]. The research in [Ara04] is particularly relevant. They developed a set of kernel modifications called TraceFS, which is a general framework for capturing and analyzing file system usage data. The modification we developed have less overhead, but are less general, than the work in [Ara04]. The work in [Ros00] describes file usage in great detail in HP UX and Windows NT environments with particular attention to the interaction between the file system and the file system cache. Our results confirm the general pattern of file system usage discovered in these previous works, though in a Linux using, web serving and GUI user interface environment. Three particular contributions of this work is to describe the directories and file types used on a dynamic basis, and to count the number of file system calls per process. We know of no other research into this area. 3. Methodology Applications can interact with the file system using a set of predefined system calls. For our research we

2 focused on (), (), open(), close(), dup(), dup2(), lseek(), (), (), (), stat(), stat64(), mmap(), and oldmmap(). We also found it necessary to record fork() and () calls in order to keep track of file descriptors and processes. Code for each of these system calls was located and modified in the Linux kernel. When each of the above system calls was invoked, our modified kernel logged data to a log file known as the file system trace. We converted each stat64() system call into a stat() system call for accounting purposes, since they are intrinsicaly the same operation. We also grouped mmap() and oldmmap(). significant web server load and is the main programming, compiling, and testing machine for our students. It has around two hundred user accounts. The trace files were processed with various Perl scripts. Program correctness was verified by two different coders. Additionally, output of the scripts was checked by hand for plausibility. 4. System Call Analysis The table below and graphs in the appendix summarize file system activity. Data was appended to this log file at the rate of about one gigabyte per week. The log file was filtered to remove any system calls related to logging. Total entries after filtering were over eighty million entries. Sample entries are shown below. FORK: PID 991 OTHERPID 1031 EXEC: PID 1031 Error 0 OPEN: fn /etc/ld.so.preload PID 1031 mode 8 flags 0 fd OPEN: fn /etc/ld.so.cache PID 1031 mode 0 flags 0 fd 3 READ: PID 1031 fd 3 count 1024 CLOSE: PID 1031 fd 3 Capturing our log data was computationally inexpensive and did not disturb the running of our servers. Typical CPU load generated by the logging was well under 1% of total CPU load. Logging our data had minimal impact on disk drive usage. Typically fifty to one hundred system calls were made by applications between each to the trace file. by our software Server slowdown was imperceptible to normal users. Because of our software design and our desire to minimize impact upon the machine, our logging software was at times overloaded with data. About 1% of total entries were lost for this reason. We have every reason to believe that the missing 1% is very similar to the 99% we captured. The modified kernel was run on three different computers. The first was a personal workstation and used to , browse the web, play music, and edit files. The second machine is the main departmental server for the computer science department. It handles a Server Workstation Call Count % Count % 6,713,519 42% 7,667,528 58% 3,323,973 21% 5,075,708 38% open 2,120,301 13% 55, % close 1,401,533 8% 55, % fork 18, % mmap 1,160, % 16, % dup 11, % 1, lseek 55, % 3, stat 967, % 323, % 8, % , % 1, , % For our departmental server four system calls dominated all activity. Read (43%), (21%), open (13%) and close (9%) constitute 86% of all system calls. Reads were twice as common as s. The average process generated 479 s and 237 s. The average opened file experienced 3.2 s and 1.6 s. We personally found it surprising that the average process generated so many file oriented system calls and that the average file was processed with so few s and s. For the personal workstation two system calls dominated all activity. Read (58%) and (38%) constitute 96% of all system calls. Open and close are not nearly as common as on the departmental server because the user of this workstation used it for tasks (long running editing session, playing music)

3 that did not generate many opens and closes. Neither compiling nor web serving was done on this personal workstation. Like the server s were almost twice as common as s. The average process generated 479 s and 237 s, similar numbers to the server. The average opened file experienced 3.2 s and 1.6 s. A file system on a personal workstation will therefore see as many s and s and s, but many fewer opens and closes than a server. Reads and s are very much larger than we expected. On the departmental server the average size of a call was 85,708 bytes. On the personal workstation it was 62,365 bytes. Writes averaged 71,584 (server) and 50,663 (workstation) bytes. Most strikiing was the difference in opens. On te web sserver 13% of all operations were opens, and on the personal workstation only 0.4% were opens. A file system optimized for web usage will emphasize open performance much more than a file system optimized for personal workstation use. 5. Most Used Directories We measured the most commonly used directories. For every file open, the directory containing that file is noted. We counted the final directory used by every file open, and not any upper level directories traversed to get to this final directory. For example, an open of /var/log/apache2/access_log was counted as an open in /var/log/apache2 but not in /var/log nor in /var nor in /. A summary is shown below. Note that for both the personal workstation and the departmental server the most common directories are /proc and /dev. Also note that the most common directories constitute a very small fraction of all file opens. The most frequently used twenty directories for our server represent only 0.3% of all file open. The most frequently used 1000 directories represent just over 3% of all file opens. For the personal workstation the equivalent fractions are only 7% and 12%. We are surprised by the conclusion there are no hot spot directories; we expected different results. A file system optimized for web server usage should expect worse performance from any directory caching, since directory operations are sp between more directories on a web server workload than on a personal workstation workload. Most directories actually used are close to the top of the directory tree. It is rare that a file is open ten levels deep, and much more common that a file is opened within two or four levels of the top of the directory tree. The many Python library directories used surprised us greatly (so much so we had to double check our work). On our computers we have a mail daemon written in Python that runs once per hour. Therefore it ran several hundred times during our logging activity. Each instance calls up several Python include files, which call up other include files, etc. We conclude that even small deamons run occasionally in scripting based languages can place a significant load on the file system, and believe that any daemon written in such a language must be written with extra care to avoid this problem. This illustrates the point that system administrators can be surprised by the results of auditing their machines. 6. Most Used File Types The chart in the appendix shows the most common file extensions for files named in open() calls. This data varies widely between server and personal workstation. Both machines used shared libraries very heavily; they were the most commonly used type of file on both systems. The Python files are discussed above. Evidence of our server s web duties is shown on this list. The *.htm and *.gif files are pages being offered to the Internet. The *.pid files are used by the Apache web server to coordinate its various processes. The workstation log data shows that the KDE desktop files (*.kde) and Python scripts used by the RedHat KDE installation (*.py, *.pyc) are heavily used. For the personal workstation the graphics files

4 (*.gif and *.JPG) are from the web browsers cache. 7. Conclusions Web servers use the file system in somewhat different ways that personal workstations. Web servers do many more file opens that personal workstations (13% of all file operations, vs 0.4%). They have a larger average size. They sp their accesses across more directories. An operating system that would optimize web performance would optimize for this different usage pattern. Understanding how applications access file systems is important in the design of both file systems and application. Using the data above we conclude that s and s are surprisingly large (around 60K). Files access is not localized to a few directories but instead is scattered throughout the file system in directories close to the root. Reads are twice as common as s and four times as common as opens. No one kind of file dominates file accesses, however shared libraries are the most commonly used kind of file. References [1] A. Aranya, C. P. Wright, and E. Zadok. Tracefs: A File System to Trace Them All. In Proceedings of the Third USENIX Conference on File and Storage Technologies (FAST 2004), San Francisco, CA, March/April [2] J. Ousterhout, H. Costa, D. Harrison, J. Kunze, M. Kupfer, and J. Thompson. A Trace Driven Analysis of the UNIX 4.2 BSD File System. In Proceedings of the 10th ACM SOSP, pages 15 24, Orcas Island, WA, December ACM. [3] J. Griffeon and R. Appleton, Reducing File System Latency Using a Predictive Approach, Conference Proceedings of the USENIX Summer 1994 Technical Conference, pages , June [4] Drew Roselli, Jacob R. Lorch, and Thomas E. Anderson, A Comparison of File System Workloads,2000 USENIX Annual Technical Conference, June 2000, San Diego CA. [5] S. Zhou, H. Da Costa, and A. J. Smith. A File System Tracing Package for Berkeley UNIX. In Proceedings of the USENIX Summer Conference, pages , June 1984.

5 System Call Distribution for Server lseek fork close 9% open 13% mmap 7% dup stat 6% 22% 43% open close fork mmap dup lseek stat System Call Distribution for Workstation 39% open close mmap lseek fork dupstat 2% 59% open close fork mmap dup lseek stat

6 Number of Opens File on Server Number of Opens File On Workstation Directory Accesses so 7,298.kde Par 3,386.so pyc 3,174 (other shared library files) py 1,581.py (other shared library files) 1,529.pyc htm 1,032.jpg gif 1,020.new o 813.slt h 805.pr loc 732.afm db 489.ali a 156.gz cla pre 145.dir all 143.gif Directory on Server Directory Access 28,670 /proc/ 27,587 /proc/ 16,176 /dev/ 15,496 /dev/ 2,093 /staff/user1/.kde/share/config/ Directory on Workstation 5,460 /staff/user1/. kde/share/config/ 1,831 /usr/share/config/ 2,351 /lib/i686/ 1,523 /etc/ 2,185 /etc/ 1,229 /usr/share/ rhn/up2date_client/ 1,741 / 1,092 /usr/sbin/ 1,699 /lib/ 1,051 /staff/user1/personal/pictures/work-backgrounds/ 801 /usr/lib/python1.5/ 976 /usr/lib/python1.5/ 715 /usr/share/rhn/ up2date_client/ Table 1: File and Directory Access Counts