Object Placement in Shared Nothing Architecture Zhen He, Jeffrey Xu Yu and Stephen Blackburn Λ

Transcription

1 45 Object Placement in Shared Nothing Architecture Zhen He, Jeffrey Xu Yu and Stephen Blackburn Λ Department of Computer Science The Australian National University Canberra, ACT fzhen.he, Jeffrey.X.Yu, Abstract Effective data placement is crucial to the performance of any parallel database system as it is an important lever for load balancing. In this paper we describe and explain a two-phased approach to the placement of objects in a shared-nothing model of a parallel object oriented database with page grain caching. The first phase is a declustering phase in which a greedy similarity graph partitioning algorithm is used to assign objects into nodes of the shared-nothing system. The aim of the first phase is to minimise internode traversals and maximise parallelism. Once assigned to nodes, the second phase attempts to cluster objects residing at each node into pages with the aim of reducing occurrence of remote page requests. Therefore, objects that are likely to be accessed by another remote node are clustered together. Both phases use query frequency distribution information to decide the placement of objects. We implemented this two phased approach on the Fujitsu AP1000 with a slightly modified OO7 benchmark data set. The results presented in this paper demonstrate that this algorithm was able to reduce remote page loads while preserving a high degree of parallelism. 1 Introduction Effective data placement is crucial to the performance of any parallel database system since it is an important lever for load balancing [Padmanadhan and Baru 92; Chen et al. 1995; Copeland et al. 1988]. In general, determining the optimal 1 placement of data across nodes of a shared nothing system 2 is a difficult problem even in relational database systems. Object-oriented database systems introduce additional complexity by supporting complex data structures which results in the object base being representable as graphs. In parallel objectoriented databases on shared-nothing architectures, the object base needs to be declustered across the nodes of the system. The manner of declustering depends on the way queries are processed. For intra-operator parallelism 3 in parallel object-oriented databases, it is important that the object placement strategy strikes a compromise between two conflicting objectives [Ghandeharizadeh et al. 1994]: a) the object placement strategy should assign all the relevant traversals by a query to a single node in order to minimise the number of inter-node object references, and b) it should distribute the workload of an application evenly across the nodes in order to maximise the processing capability of the system. These objectives are in conflict because a) argues for assigning objects to a few nodes in order to group the relevant objects together, while b) advocates the distribution of the objects across as many nodes as possible. 2 Similarity Graph Partitioning We have based our object placement strategy on a similarity graph partitioning approach. Fang, Lee, and Chang [1986] introduced similarity graph partitioning to decluster data across multiple disks of a single processor multiple disk system. Liu and Shekhar [1996] proposed a variation of Fang, Lee, and Chang [1986] s approach in which query frequency distribution information was used to define similarity. They proposed a general approach to declustering which they applied to grid files of a spatial database. Their aim was to maximise the chances that a pair of atomic Λ The authors wish to acknowledge that this work was carried out within the Cooperative Research Center for Advanced Computational Systems established under the Australian Government s Cooperative Research Centers Program. 1 optimal in terms of performance 2 each node has its own memory, disk and CPU 3 one operator is processed across all the nodes of the parallel system. Typically, each node works on its own disjoint subset of the problem and its own data set. Parallelism is dependent on the way data items are positioned.

2 46 data-items that are frequently accessed together by queries, are allocated to distinct disks. They defined similarity between two items as the weight of the edge joining two data items. The weight is assigned as the likelihood that the pair of data items will be accessed together by queries in the query set of interest. Therefore they place data items which are more similar into distinct nodes. 3 Preliminaries In this paper, we consider intra-operator parallelism for a parallel object-oriented database system on a shared-nothing architecture where each node has its own memory, disk and CPU. Page grain caching is also incorporated into the model. We focus on fully declustering which declusters objects across multiple nodes where each object only has a single copy in the object base. In addition to reduced cache misses for requests by other nodes, we cluster onto the same page objects held at each node which have, similar remote access characteristics. The paper is concerned with offline object placement methods. Therefore the objects are placed while the database in not in operation. In such an environment we seek to find the object placement method which minimises the total execution time of an arbitrary set of queries. 4 Declustering We used a variation on the similarity based approach proposed by Liu and Shekhar [1996] to decluster the objects in the data set. Our algorithm differs from Liu and Shekhar [1996] in that we attempt to place objects that have a higher degree of similarity together in the disk of the same node in contrast to their goal of placing similar data items on different disks. Our definition of similarity states that two objects are more similar if they are accessed together in a navigational manner but less similar if the two objects can be accessed together in a parallel manner. This enables the algorithm to preserve a degree of parallelism while decreasing internode object references. The degree of declustering similarity of an object o i with a partition P j of objects is defined as follows: S d (o i ;P j )=α w link (o i ;o k )+w link (o k ;o i ) (1 α) w p (o i ;o k ) (1) o k 2P j o k 2P j where α is a user defined weight that is assigned based on the relative importance of internode object references in reducing response time as compared to parallelism. The term w link (o i, o j ) is used to denote the frequency of traversal of the pointer link from o i to o j. w p (o i, o j ) denotes the frequency with which the two objects o i and o j can be processed on different nodes in parallel. Two approaches were used to decluster the database, the first is a greedy approach and the second is a window approach. The greedy approach first sorts all objects to be placed into an array based on non-increasing frequency of reference. Then each object o i is placed one at a time into the partition P j which has the largest declustering similarity value with the object o i. In the case where a tie occurs, the object is placed in the partition on which the least number of ties have been assigned. The window approach differs from the greedy approach only in the order in which objects are considered for placement. In the window approach objects are considered in order of reachability. Two hash tables were used to efficiently calculate the similarity between an object and a partition. 5 Clustering The clustering phase is composed of two steps. The first step clusters objects of the same type together. This benefits intra-operator parallelism since each node iterates through the portion of objects that reside in its own disk. Once clustered together based on type, the second step reduces remote page loads by clustering objects which are likely to be accessed by another node together. The implementation of the first step is straight forward but the second is not as obvious. Our approach is to partition the objects to be clustered at a particular node into N partitions where each partition is reserved for a node. Each object is assigned to the partition which has the largest similarity value with respect to the node that the partition is clustered for. This requires a slightly different definition of similarity. The degree of clustering similarity of an object o i with a partition P j of objects is defined as follows: S c (o i ;P j )= w link (o i ;o k )+w link (o k ;o i ) (2) o k 2P j

3 47 We reuse the two hash table technique developed for the declustering process to calculate the clustering equation efficiently. 6 Query Processing Four different queries were tested on the objects of a varying sized OO7 benchmark [Carey et al. 1993]. Varying the size of the OO7 benchmark allowed us to test the scalability of our algorithm by observing response time as the number of processors increase in proportion to problem size. As a base case a slightly modified medium sized OO7 benchmark was used for 32 nodes. ffl Query-1 (a range query): The first query is a set based query which counts the number of atomic parts with build date above ffl Query-2 (a value-based join query): The second query is based on query 8 of the OO7 benchmark. The query involves finding all pairs of documents and atomic parts where the document ID in the atomic part matches the ID of the document [Carey et al. 1993]. The query is processed by performing a value based hash join. ffl Query-3 (a navigational query with sharing): The third query is based on query 5 of the OO7 benchmark. Query 5 of the OO7 benchmark involves finding all base assemblies that use a composite part with a build date later than the build date of the base assembly. ffl Query-4 (a navigational query without sharing): The final query involves finding all composite parts with build dates earlier than its atomic parts build date. We call Query-1 and Query-2 set queries, and call Query-3 and Query-4 navigational queries. An important point to note is that there exists conflicts between the different queries. For Query-1 and Query-2, atomic parts need to be evenly placed on the nodes to allow a higher degree of parallelism but for optimal placement of atomic parts, Query-4 requires atomic parts to be placed together in order to decrease the number of internode traversals. Similar conflicts exists for Query-3 and Query-4. 7 Overview of Test Bed The hardware platform used is the 128 node Fujitsu AP1000 [Ishihata et al. 1990]. Each node has a 25 MHz SPARC CPU with 16MB of memory and 128 KB of direct-mapped cache memory. There are 16 disks in the system. We have simulated a shared-nothing system on this hardware architecture. Each node of the system acts as both a client and a server. The clients do the query processing and the server is responsible for satisfying remote page requests. Initially the system provided two limitations in providing our desired test bed. The first limitation is a lack of support for threads by the operating system and the second is the parallel file-system. The two limitations where overcome by the use of a MPI [Sitsky et al. 1994] non-blocking receive and an air-disk respectively. 8 Performance Results The four queries outlined in Section 6 were first run to train the database, therefore to arrive at the values for the frequency counts. The frequency counts were then used by the various placement algorithms to position the objects into nodes. The four queries were then run 10 times with the objects placed according to the various algorithms to obtain the results presented. When a query completes, the cache is flashed thus all results presented in this paper are derived from cold runs. Note: The labels on the graphs mean the following: ffl RD : Random partitioning ffl RR : Round Robin partitioning ffl DN : Greedy similarity declustering without similarity clustering ffl DC : Greedy similarity declustering with similarity clustering ffl WIN: Window similarity declustering with similarity clustering

4 Respnose Time (sec) RD RR DN DC WIN Number of processors Figure 1: Response time for running all four queries The value of α for DN, DC and WIN were all set to 0.9 since the queries run involved only a small amount of computation. The complete set of results for this project are ommitted from this paper due to paper length limitations, however they can be found in He [1998]. The results on figure 1 show the placement algorithms developed in this paper performing only slightly better than random and round robin. The reason is the execution time of query 2 (parallel value based join) dominates the total execution time of running all four queries. Query 2 can not be optimised by the placement algorithms since it is a set query which does not benefit from reduced navigation. The fact query 2 involves all to all communication and synchronisation also causes the total execution time to appear very non-scalable. This however is not the case for queries 3 and 4 where navigation is present. Queries 3 and 4 when placed via WIN depicted execution times upto 8 fold lower than random placement. WIN appears to be the best performer when navigation is present in the query. From the experiments conducted it was also observed WIN and DC resulted in large reductions (upto 16 fold) in the number of remote page loads while causing only a small amount of client workload imbalance. Another experiment we conducted involved varying training conditions 4 and execution conditions. The results showed the clustering phase was essential in reducing remote page loads when the training conditions differ vastly from execution conditions. This is due to the fact the clustering phase do not need to balance between parallelism and reduced internode object references. 9 Conclusion Both the clustering and declustering of objects in a parallel object oriented database is important in reducing the response time of a set of queries that contains navigation. The fact that the declustering plus clustering approach resulted in a large reduction in the number of remote page loads while causing only a small amount of client workload imbalance is grounds for optimism. The extension of these ideas to online 5 object placement would be an interesting area for further research. Testing with a larger set of queries with more conflicting requirements is important in predicting the performance of the object placement methods in real applications and thus should be explored as a part of further research. Acknowledgement We would like to thank Luke Kirby for helping in the implementation of the test bed and for providing helpful comments and suggestions, David Sitsky and David Walsh for their help in the implementation of the test bed, and Richard Walker for typographical assistance. 4 queries broken up into two categories and they include set and navigation queries. Varying percentage of set and navigational queries where tested. 5 while the database is in operation

5 49 Bibliography CAREY, M., DEWITT, D.,AND NAUGHTON, J The OO7 benchmark. In Proceedings of the 1993 ACM- SIGMOD Conference on the Management of Data. CHEN, L.T.,ROTEM, D.,AND SESHADRI, S Declustering databases on heterogeneous disk systems. In Proceedings of the 21st VLDB Conference (1995), pp COPELAND, G.,ALEXANDER, W.,BOUGHTER, E., AND KELLER, T Data placement in bubba. In Proceedings of the 21st VLDB Conference (1988), pp FANG, M., LEE, R., AND CHANG, C The idea of declustering and its applications. In Proceedings of twelfth International conference on Very Large Databases, GHANDEHARIZADEH, S.,WILHITE, D.,LIN, K., AND ZHAO, X Object placement in parallel objectoriented database systems. Proceedings of the Tenth International Conference on Data Engineering, HE, Z Object placement in parallel object oriented databases. Honours Thesis. Australian National University. ISHIHATA, H.,HORIE, T.,INANO, S.,SHIMIZU, T.,AND KATO, S CAP-II architecture. In Proceedings of the First Fujitsu-ANU CAP Workshop (Kawasaki, Japan, November 1990). Fujitsu Laboratories Ltd. LIU, D.AND SHEKHAR, S Partitioning similarity graphs: A framework for declustering problems. Information Systems 21, 6 (September), PADMANADHAN, S.AND BARU, C. K. 92. Data placement in shared-nothing parallel database systems. Technical report, The University of Michigan. SITSKY, D.,WALSH, D.,AND JOHNSON, C An efficient implementation of the message passing interface (MPI) on the Fujitsu AP1000. In Proceedings of the Third Parallel Computing Workshop (Kawasaki, Japan, Nov ). Fujitsu Laboratories Ltd.

6 50