A Framework for Storage Management Evaluation in Persistent Object Systems

Transcription

1 A Framework for Storage Management Evaluation in Persistent Object Systems Thorna O. Humphries Alexander L. Wolf Benjamin G. Zorn University of Colorado University of Colorado University of Colorado Abstract With the advent of object oriented database management systems, the need to evaluate and improve the performance of these systems has led to research in the area of system benchmarking. One traditional approach has been to implement an application using a particular database management system and incorporate methods in the application to perform analysis and experimentation on functions of the system, such as storage management. In this paper, we present another approach to performance evaluation. We describe PTFF, a common trace file format, and Tragedy, a trace generation tool. The advantages of our approach include: Applications can be modeled independent of any database management language or system. The instrumentation necessary to perform experimentation and analysis of storage management is transparent to the application layer. The trace event files only have to be generated once and can be used for several experiments. The development of applications for system benchmarking can be reduced through the reuse of classes provided as libraries. The experiences of the POSSE group with our previous approaches are also discussed as well as our newest approach. 1. Introduction In the area of object database management system (ODBMS) development, there is a need to have benchmarks for assessment of their performance. The POSSE (Persistent Object SyStem Evaluation) project seeks to design, implement, and evaluate alternative implementation methods for persistent object systems 1. For the past three years, our work is specifically focused on the problem of storage management in such systems [CWZ 94, CKWZ 96, CWZ 98]. In our efforts to develop new policies for storage management, we have developed a trace file format, PTFF [KWZ 96], and hand instrumented an implementation of the OO7 benchmark [CDN 93] with methods to record its structure and behavior using that format. To evaluate or improve performance of a system, its behavior must be captured in a meaningful manner. Conceptually, the process of benchmarking consists of three elements: the behavior of an application, a representation of the behavior, and the 1 For more information about the POSSE project, visit our WWW page at: 1

2 analysis with respect to some aspect of the performance of the object database system. Figure 1 shows the elements of the benchmark process and examples of these elements. In order to capture the behavior of an application, the application must be instrumented so that the relevant events can be recorded. For example, in the case of storage management, the creation of objects, the deletion of objects, and the creation of edges from a given object are typical events that must be recorded to obtain a representation of the structure of the database and the behavior of the database application. In this paper, we will discuss our experiences with the development of traces using the OO7 benchmark and the Exodus storage manager for storage management analysis. The lessons learned from this endeavor have led us to examine other approaches to instrumentation of object database applications for system benchmarking. The main focus of this paper is to describe one alternative approach that uses Tragedy (TRAce GEneration for Database systems) [HWZ 97] and the POSSE trace file format (PTFF) [KWZ 96] to capture the structure and application behavior of object database applications. Process Elements Examples Applications + Instrumentation OO7 Tragedy Trace File PTFF file Analysis Tools ODBsim, Visualization, Statistics (SPSS, S+) Figure 1. Elements of System Benchmarking PTFF is the specification of a set of events that characterize the operations on object databases and persistent stores. It provides a means of capturing the structure and the behavior of an object database in a system and machine independent way. PTFF files can be used for a variety of investigations, including simulation studies, application visualizations, debugging, and statistical summaries of application behavior. We 2

3 generated traces in PTFF format in our hand instrumented E/Exodus implementation of the OO7 benchmark. Tragedy is a tool that consists of a TCL interface and a collection of classes to allow the transparent instrumentation of a database application. The TCL interface provides a interactive mechanism for setting up the activities of the benchmark, such as database creation, the execution of traversals and the execution of queries. The collection of classes of this tool provides a mechanism for instrumenting the operations of a database application. With Tragedy, we are able to model object database applications with no database management language or system support, with transparent support for generating our trace files, and with high-level support for constructing complex application behaviors. The remainder of this position paper is organized as follows. In the next section, we discuss our previous experiences with the instrumentation of the OO7 benchmark. We also discuss related work. A discussion of PTFF is presented in Section 3. Section 4 presents Tragedy. We conclude in Section 5 with a discussion of the status of our work and plans for future work. 2. Background The POSSE group is interested in developing and evaluating high performance algorithms for storage management in object database systems. Our research has concentrated on partition-based algorithms, and our previous results include new methods for partition selection [CWZ 94] and methods for controlling garbage collection rates [CKWZ 96]. Our approach to performance evaluation has evolved in the last several years, and the interfaces and formats described in this paper are the result of our experiences. Our initial approach to performance evaluation involved simultaneously developing a simulation system and synthetic database application. The application, ABT (Augmented Binary Tree), was a parameterized synthetic benchmark that interacted via procedure calls directly with our ODBMS simulation system (ODBsim). This monolithic approach had weaknesses, including the fact that changing the application meant rewriting code, and understanding the system was difficult. Because we wanted to relate our performance results to more realistic database applications and previous work in ODBMS storage management, we chose to use OO7 as a benchmark, which meant we had to restructure our simulation system. Our secondgeneration approach followed the process illustrated on the left-hand side of Figure 2. In particular, the original OO7 benchmark was hand-instrumented to produce an instrumented version of the application, and this version was run in the context of the Exodus storage manager to produce a trace file. We also modified our simulation system, ODBsim, to read the specified trace format. 3

4 Application Application Spec. Tragedy Lib. Schema Instrument By Hand Instrument Using Tragedy Instrumented Application Instrumented Application DBMS EXECUTE EXECUTE Trace File Analyzer Perf. Results Figure 2. Two Approaches to Instrumentation and Evaluation of an ODBMS 4

5 While the second generation was an improvement over our first generation technology, there were still problems. In particular, instrumenting the OO7 application required modifying the original code to add trace-gathering instrumentation, and instrumenting OO7 gave us no additional leverage when we wanted to instrument a new ODBMS application. Furthermore, the OO7 implementation we were using was developed on a MIPS-based Decstation, and in the subsequent years, our local MIPS machines were replaced by DEC Alphas, rendering the earlier instrumentation unusable. The result of this previous experience is the combination of Tragedy and PTFF, as illustrated on the right-hand side of Figure 2. In this case, the structure of the application database is modeled using predefined Tragedy library classes representing objects and edges, and the behavior of the database application is specified using a combination of high-level phase and traversal specifications in TCL, and a lower-level implementation specified in C++. These components are integrated with the Tragedy library, and an instrumented version of the database application results. 2.1 Related Work Researchers in the area of memory storage management have effectively used the tracedriven approach to instrument systems for performance evaluation. This approach has been used in the studies of malloc algorithms such as in [DDZ 94, KV 85]. In order to capture various types of information about the behavior of a program or system, several trace formats have been developed. In the area of database systems, the MaStA I/O trace format [SCM+ 95] was developed to capture information necessary for performance evaluation of I/O costs. The trace events of MaStA record information on reading, writing, and synchronization operations that are performed on a database. Recently, a trace-driven approach has been taken to develop a prototype program visualization system, PV [ KRR 94], at IBM Research. The trace format of the PV system records the event type identifier, a timestamp, and some event-specific data for each of its supported events. Some of the events of this system include a page fault, allocation of a region of memory, and receipt of a message. 3. POSSE Trace File Format (PTFF) The goals of our work on a common trace file format include the following: To develop a trace format that can be used to study and evaluate performance issues such as storage management, well as other analysis activities. To develop a system independent representation of object database application behavior. To support comparative experimentation of ODBMS implementations by promoting the creation of a collection of representative object database application behaviors in a common file format. 5

6 The POSSE trace file format consists of a collection of events that capture the structure and application behavior of a database system. The structure of an object database is captured through the object creation, the edge creation, and the data writing events. Its application behavior is captured through the edge reading, the data writing, and the data reading events. Examples of the events that are represented in PTFF are shown in Figure 3. PTFF trace files can be used as input to a variety of analysis tools. An example of an analysis tool using PTFF files is ODBsim [CWZ 93], a simulator that the POSSE group uses to experiment with different object database storage management policies. Trace Begin Co Dw Dw Ew Dr Sr 10 // object creation with 11 edges // write data at offset 0 of the object for a length of 4 bytes // write data at offset 4 of the object for a length of 4 bytes // write edge number 0 and connect it to the null object // read data at offset 0 of the object for a length of 4 bytes // make the object a persistent root Figure 3. Examples of Trace File Events represented with PTFF 4. Tragedy Trace Generation Tool 4.1 Overview of Tragedy The trace generation tool was conceived in an attempt to synthetically generate a collection of object databases that are representative of real databases and their applications. It consists of two components, a TCL interface and a collection of library classes to model and instrument object database applications. The results of executions are stored in PTFF files. The goals of Tragedy are as follows: To provide a flexible, high-level toolkit for the construction of diverse and complex object database application behaviors. To allow transparent instrumentation of application behavior and generation of a common trace file format (PTFF). To synthetically generate a collection of object database systems. To generate event traces that reflect an application s behavior. The TCL interface is provided to aid the user in developing the activities of the benchmark. The database application is viewed as consisting of phases, which may consist of one or more of the following activities: a database creation activity, a database re-organization activity, and an activity in which the data is manipulated through traversals, queries, and updates. Each phase is supported by the TCL interface through the Tragedy application and user-defined TCL commands. 6

7 Tragedy consists of a collection of classes, some of which are used to model the objects of the database system. These classes are GraphNode and Edges. The class GraphNode is used as a base type for the objects of the database system. The class Edges is used to model references contained in these objects. Through the classes provided by Tragedy, trace events are automatically generated as operations are applied to the objects of the database system. These trace events are then recorded in a file using PTFF. 4.2 Modeling a Database Application In order to use Tragedy, the schema and the application specification of a database system must first be converted into objects that are implemented as C++ classes. These objects become subtypes of the GraphNode object. They are modeled as graph nodes with directed edges. In Figure 4, the data structure for the object CompositePart, which has been extracted from the C++ version of the OO7 benchmark [CDN 93], is modeled using the objects, GraphNode and Edges. The object CompositePart is created as a subtype of the object GraphNode. Also, the pointer values of the CompositePart class definition become edges when they are represented by the data structures of Tragedy. The methods of the database application that implement queries and updates are modeled as traversal and operation objects. The application interface provides three predefined traversal objects. These traversal objects implement a depth-first traversal with pre-node processing, a depth-first traversal with post-node processing, and a breadth-first traversal. These traversal objects are implemented so that the operation performed at each object is generic. Operations can perform a variety of tasks, such as updating an object, referencing the data information of an object, or updating the structure of the database by adding or deleting an object, and are determined by the database application. Some of the methods are then mapped into TCL commands. The TCL interface of Tragedy provides commands to create the database and build the pre-defined traversals. User-defined commands that invoke methods that define operations on the objects of the database can be added to the TCL application layer. 5. Conclusion Tragedy provides a framework for developing event traces that capture the structure as well as the behavior of synthetically generated applications. We feel that instrumentation of an application on top of a database management system limits the generation of a trace file to the particular platform that supports the database management system or object storage system. We also found that the process of instrumentation that requires a translation of the database schema to a particular database language can be a very time consuming endeavor and cannot be altered quickly to gather other information about the application. 7

8 Schema for DesignObj and CompositePart Class DesignObj{ Int4 Char Int4 }; id; type[10]; builddate; class CompositePart: DesignObj { Document* Bag(BaseAssembly*) Bag(BaseAssembly*) Set(AtomicPart*) AtomicPart* } with extent (id indexed); documentation; usedinpriv; usedinshar; parts; rootpart; Composite Part GraphNode DesignObj Trace Data Object Pointer OutEdges Id Type builddate Edges Documentation UsedInPriv UsedInShar Parts RootPart Figure 4. Representation for object CompositePart in OO7 Benchmark 5.1 Status Tragedy is implemented in C++. The tool provides an interface for recording trace events. It also provides a TCL front-end that allows a user to specify various parameters that are needed to generate the database and then to specify traversals and queries to be performed on the objects of the data base system. The trace generation tool has been tested using a few synthetic single user workloads. The traces have then been submitted to our in-house simulator to be processed. The tool has also been used to model the inmemory version of the OO7 benchmark. We are currently in the process of comparing 8

9 the trace files generated with Tragedy to those generated through instrumentation of the OO7 benchmark using Exodus. At the time of this writing, we have not concluded our evaluation. 5.2 Future Work The trace generation tool is currently set up to generate workloads based on a single user. Although the single user workload provides useful information for some forms of analysis, it is not completely representative of workloads of real database systems. In order to create a more realistic modeling tool, one of the enhancements that will be made to the tool is to incorporate transaction processing and locking support to allow for concurrent access. We would also like to model a portion of a real object oriented database application to determine the ease of modeling such an application using the trace generation tool. In addition to the above enhancements, we would like to expand our trace file format by adding additional events to handle dynamic edges and dynamic data. Furthermore, we are looking at using the trace file produced by Tragedy to visualize the time varying behavior of the database structure. Acknowledgments This work was supported in part by the National Science Foundation under grant IRI References [CDN 93] Michael J. Carey, David J. DeWitt, and Jeffrey F. Naugton. The OO7 Benchmark. In Proceedings of the ACM SIGMOD International Conference on the Management of Data, pages 12 21, Washington, DC, June [CWZ 93] Jonathan E. Cook, Alexander L. Wolf, and Benjamin G. Zorn. The design of a simulation system for persistent object storage management. Technical Report CU-CS , Department of Computer Science, University of Colorado, Boulder, CO, March [CWZ 94] Jonathan E. Cook, Alexander L. Wolf, and Benjamin G. Zorn. Partition selection policies in object database garbage collection. In Proceedings of the ACM SIGMOD International Conference on the Management of Data, pages , Minneapolis, MN, March [CWZ 98] Jonathan E. Cook, Alexander L. Wolf, and Benjamin G. Zorn. A highly effective partition selection policy for object database garbage collection. To appear in IEEE Transactions on Knowledge and Data Engineering. [CKWZ 96] Jonathan E. Cook, Artur W. Klauser, Alexander L. Wolf, and Benjamin G. Zorn. Semiautomatic, self-adaptive control of garbage collection rates in object databases. In Proceedings of the ACM SIGMOD International Conference on the Management of Data, pages , Montreal, Canada, June

10 [DDZ 94] David Detlefs, Al Dosser, and Benjamin Zorn, Memory Allocation Costs in Large C and C++ Programs, Software-Practice and Experience, pages , June [HWZ 97] Thorna O. Humphries, Alexander L. Wolf, Benjamin G. Zorn. Tragedy, Trace Generation Tool Design Specification. Department of Computer Science, University of Colorado, Boulder, CO. (in preparation) [KRR 94] Doug Kimelman, Bryan Rosenburg, and Tova Roth. Strata- Various: Multi-layer visualization of dynamics in software system behavior.[ On-Line] Available URL: [KWZ 96] Artur G. Klauser, Alexander L. Wolf, and Benjamin G. Zorn. POSSE Trace File Format Version 0.9 Draft. Department of Computer Science, University of Colorado, Boulder, CO, June [KV 85] David G. Korn and Kiem-Phong Vo, In Search of a Better Malloc, Proceedings of the Summer 1985 USENIX Conference, pages , [SCM+ 95] S.J.G. Scheuer, R.C.H. Connor, R. Morrison, J.E.B. Moss and D.S. Munro. The MaSta I/O trace format. School of Mathematical and Computational Sciences, University of St. Andrews, North Haugh, St Andrews, Fife, Scotland. Technical Report CS/95/4,