Middle Layer Java Software for the Implementation of a Homogeneous Distributed Database System

Transcription

1 Middle Layer Java Software for the Implementation of a Homogeneous Distributed Database System Created By, Sanjib Kumar Das, 03CS1018 Sayantan Dasgupta, 03EE3201 DBMS LAB 19 th April, 2007

2 2 Contents 1. Why Distributed Database System Unsolved Problems of DDBMS Features of JSQL Query Execution Conclusion Reference

3 3 1. Why Distributed Database System A distributed database is a collection of multiple, logically interrelated databases distributed over a computer network. And, a distributed database management system (distributed DBMS) is the software system that permits the management of the distributed database and makes the distribution transparent to users. fig: Distributed Database Management Architecture In this section, we consider the commonly cited advantages of distributed DBMS. 1.1 Transparent management of distributed and replicated data Distributed database technology is intended to extend the concept of data independence to environments in which data is distributed and replicated over a number of machines connected by a network. Data independence is provided by several forms of transparency: 1. Network (and, therefore, distribution) transparency. 2. Replication transparency, and 3. Fragmentation transparency. 1.2 Reliability through distributed transactions Distributed DBMS are intended to improve reliability, since they have replicated components and thereby eliminate single points of failure. In a distributed database, this 3

4 4 means that some of the data may be unreachable, but with proper care users may be permitted to access other parts of the database. Distributed transactions execute at multiple sites, where they access the local database. With full support for distributed transactions, user applications can access a single logical image of the database and rely on the distributed DBMS to ensure that their requests will be executed correctly no matter what happens in the system. 1.3 Better performance Distributed DBMS superior performance is usually based on 2 points: 1. Data localization: Fragmentation of the conceptual database, enabling data to be stored in close proximity to its points of use. This feature has 2 advantages : Since each site handles only a portion of the database, contention for CPU and I/O services is not as severe as for centralized databases. Localization reduces remote access delays and also reduces costs. 2. Inherent parallelism of distributed systems can be exploited for inter-query and intra-query parallelism. Inter-query parallelism: execution of multiple queries at the same time Intra-query parallelism: breaking up of a single query into a number of subqueries, each executed at a different site, accessing a different part of the distributed database. 1.4 Easier, more economical system expansion In a distributed environment, accommodating increasing database sizes should be easier. Major system overhauls are seldom necessary; expansion can usually be handled by adding processing and storage power to the system. This is known as database size scaling. 4

5 5 2. Unsolved Problems in DDMBS Distributed database technology is one of the most important computing developments of the past decade. During this period, distributed database research has been intense, culminating in the release of a number of first-generation commercial products. But yet, there are a few problems that Distributed Systems cannot solve. 2.1 Network scaling problems Questions have been raised about the scalability of some protocols and algorithms as systems become geographically distributed or as the number of system components increases. The proper way to deal with scalability is to develop general, sufficiently powerful performance models, measurement tools, and measurement methodologies. 2.2 Distribution design Distributed database design methodology varies according to system architecture. Top-down design: From requirements analysis and logical design of the global database to physical design of each local database. Employed for tightly integrated distributed databases. Bottom-up design: Involves the integration of existing databases. These are employed for distributed multi-database systems. Two aspects of distribution design are: 1. Fragmentation 2. Allocation Fragmentation is the partitioning of each global relation into a set of fragment relations. Allocation concentrates on the distribution of these local relations across the distributed system s sites. Allocation is typically treated independently of fragmentation. However, closer examination reveals that isolating the two steps actually contributes to the complexity of the allocation models. Combining these two steps may have synergistic effects enabling the development of acceptable heuristic solution methods. 2.3 Distributed query processing Distributed query processors automatically translate a high-level query on a distributed database, which is seen as a single database by users, into an efficient low level execution plan expressed on the local databases. There are two important requirements of such a translation: 1. The translation must be a correct transformation of the input query. 5

6 6 2. The execution plan must be optimal that is, it must minimize a cost function that incorporates resource consumption. Thus the query processor must investigate equivalent alternative plans to select the best one. Global query optimization is typically performed prior to execution of the query (hence, called static optimization).a major problem with this approach is that the optimization cost model may become inaccurate because of changes in fragment size or because of database reorganization, often done for load balancing. 6

7 7 3. Features of JSQL 3.1 Homogeneous distributed database: All sites would have identical database management system software.we have opted for MySQL. In a heterogeneous database, different sites may use different schemas, and different database management system software. Our project is aimed at demonstrating the basic concepts employed in a distributed database and so we decided to avoid the hindrances caused due to divergence in software. 3.2 Data Storage: The 2 standard approaches generally employed for storing a relation in a distributed database are: 1. Replication: Several identical copies of the relation are stored at different sites. 2. Fragmentation: The relation is partitioned into several fragments, and each fragment is stored at a different site. Horizontal fragmentation: The relation r is partitioned into a number of subsets r1, r2,...rn. Some tuples of the relation are kept at one site and the other tuples are distributed among the other sites. Vertical fragmentation: The set of attributes R of the relation r is partitioned into a number of subsets. So here each site contains all the tuples of a relation but not all the attributes. In our project we have decided to employ an approach different from these standard approaches. Fragmentation would be such that there will be no replication of data among sites. Replication has the advantages of availability and increased parallelism but it also leads to increased overheads of storage and update. Different sites will have the different tables of the schema. We are neither going for horizontal fragmentation nor vertical fragmentation. In case of horizontal fragmentation, all the queries of the user would have to be sent to all the sites. The result sets retrieved would be mutually exclusive and so taking union of these sets would be the final result set. The work would have been very trivial! In case of vertical fragmentation the work would be a little heavier! In our scheme, complete tables are kept in a site, but not all the tables in the same site. The plan is to keep related tables in one site. 7

8 8 Consider a banking enterprise (given in the text book) having the following relation schemas: branch (branch_name,branch_city,assets) customer (customer_name,customer_street,customer_city) loan (loan_number,branch_name,amount) borrower (customer_name,loan_number) account (account_number,branch_name,balance) depositor (customer_name,account_number) We could distribute the tables among 3 sites: Site 1: branch Site 2: customer, borrower, depositor Site 3: loan, account There will be a central data dictionary which will contain the information regarding which tables are in which sites and the schema definition of the tables. In our distribution of the database, apparently allocation and fragmentation seem to be independent. But it is not exactly so. We have looked into the various example queries in the text book which has helped us in guessing the pattern of general queries on this database. We have tried to put related tables in one site. 3.3 Architecture of the system: We plan to build middle-layer software (chiefly a Query Processor) consisting of 3 main components: 1. Query Decomposer: It would parse the query fired by an user and break it into sub queries and migrate the queries to the relevant sites. While doing so, it would refer to the central data dictionary mentioned earlier. The exact way in which this parser would work would be clear by the examples to follow in the next section. 8

9 9 fig: Stages of Query Executions This is how a query decomposer, in general, looks like. In our case, data localization and global optimization are not kept separate. When decomposing the query into fragmented queries we are trying to minimize the cost. Local optimization is being taken care of by the local MySQL DBMS. The Global Schema and the Local Schema are both contained in one file, which we are calling the central data dictionary. 2. Result joiner: It would join the result sets obtained from the different sites. Optimization techniques would be applied during the joins. We also plan to implement pipelining (parallelism during the joins) i.e. suppose we have to join 4 relations r1,r2,r3 and r4,we could join (r1,r2) and (r3,r4) parallel. However, with respect to the example bank database at most 3 result sets can be obtained. So here this parallelism could not be employed. We have implemented the Mergejoin algorithm in our project. 3. Authentication system: This would realize the basic login-logout operations. The users must first login onto a server in which the middle layer software runs, in order to access the databases, and middle layer software communicates to individual databases. This would help the software to map the queries to the corresponding users. Whenever the login is authenticated, the following GUI appears before the user. 9

10 10 fig: GUI for entering queries Here we have to consider concurrency control in more details when the various users try to access the same database. This is achieved by the sessions. A session belongs to a user. The various queries are mapped to the respective sessions. These way results of queries fired by different users do not get mixed up. Concurrency control on the different databases is being taken care of by the respective MySQL database servers. 10

11 11 4. Query Execution Let us illustrate the working of the Query Parser using some examples. 1. Queries involving tables in a single site are trivial. These queries would remain unchanged and would be executed at the relevant site. The results retrieved would be passed to the user. Now consider a query: select customer_name,borrower.loan_number,amount from borrower,loan where borrower.loan_number = loan.loan_number and branch_name = Perryridge and amount > 1200 Now borrower is in site2 and loan is in site3. One straightforward strategy would be to move the tables borrower and loan to the the middle-layer software and execute the query there.but this would mean migration of large datasets which is very inefficient.so our plan is to break the query into 2 subqueries and executing these subqueries at the relevant sites and then migrating the smaller data sets obtained as results. Site2 : select customer_name,borrower.loan_number from borrower Site3 : select loan.loan_number from loan where branch_name = Perryridge and amount > 1200 The results obtained from these 2 sites would then be moved to the result joiner part of the Query Processor where the join would be performed by running the query : select customer_name,borrower.loan_number from borrower,loan where borrower.loan_number = loan.loan_number Let us formalize this algorithm for decomposing queries. Example query : select customer_name,borrower.loan_number,amount from borrower,loan where borrower.loan_number = loan.loan_number and branch_name = Perryridge and amount >

12 12 Steps of Query Execution: Step 1: Create 3 lists : SelectItems, FromItems, WhereItems. Main operator in the where part of the query is also determined. These will contain the respective items in the global query. Here, SelectItems : {customer_name,borrower.loan_number,amount} FromItems : {borrower,loan} WhereItems : {(borrower.loan_number=loan.loan_number), (branch_name= Perryridge ), (amount>1200) } Main Operator : and Step 2: Look at the FromItems and refer to the central data dictionary to determine which sites are to be considered for this particular query. Create n number of SubSelect, SubFrom and SubWhere lists where n : no. of sites Distribute the SelectItems into the SubSelect(n) lists, FromItems into the SubFrom(n) lists. Example : FromItems : {borrower,loan} borrower -- site 2 loan --- site 3 SubFrom[1] = {}-empty SubFrom[2] = {borrower} SubFrom[3] = {loan} SubSelect[1] = {} SubSelect[2] = {customer_name,borrower.loan_number} SubSelect[3] = {amount} If SubFrom[i] is empty, no need to consider its SubSelect and SubWhere. Step 3: Pick each item from the Where-Items list. If that clause belongs completely to a site, then include that clause into the corresponding SubWhere list. If a clause does not belong completely to one site then put that clause into a new list Final Where. 12

13 13 Parse that clause to get the operands. Look for attributes of tables in individual operands. Find the sites to which these attributes belong to. If these attributes are not included in the respective Sub Select then include them. This step is one of the key steps in our algorithm. It is not intuitive and the example would help us understand it. Example : 1 st item from Where Items : borrower.loan_number=loan.loan_number Looking at the operands(borrower.loan_number,loan.loan_number) we find that this clause refers to two sites. FinalWhere = {( borrower.loan_number=loan.loan_number)} borrower.loan_number is there in SubSelect[2]. loan.loan_number is not there in SubSelect[3].So include it. Now, SubSelect[3] = {amount, loan.loan_number} This is very important.otherwise we would have lost this column in the result set obtained from site 3 and the final query would never have taken place. 2 nd item from WhereItems : branch_name = Perryridge This clause belongs completely to site 3 since branch_name is an attribute of loan relation. SubWhere[3] = {(branch_name = Perryridge )} 3 rd item from WhereItems : amount >1200 This clause again belongs completely to site 3 since amount is an attribute of loan relation. SubWhere[3] = {(branch_name = Perryridge ),(amount>1200)} Step 4 : Generation of subqueries and final query Using the SubSelect,SubFrom and SubWhere lsits generate the subquery. FinalSelect = SelectItems FinalFrom = {r1,r2..,rn) where n : no of sites and rn stands for the result set obtained from the nth site. Generate the final query from the FinalSelect,FinalFrom and FinalWhere lists. Example : Subquery[1] =nil SubQuery[2] : select customer_name,borrower.loan_number from borrower SubQuery[3] : 13

14 14 select amount, loan.loan_number from loan where branch_name = Perryridge and amount>1200 FinalQuery : select customer_name, borrower.loan_number, amount from r2,r3 where borrower.loan_number = loan.loan_number where r2,r3 are result sets obtained from sites 2 and 3 respectively. Step 5: Execute the sub queries at the respective sites parallel (intra-query parallelism) and the final query at the middle layer. This final query execution is done by the Result Joiner part of our middle layer software,jsql. For joining the result sets, we employed the strategies that are discussed in the Joining of two Relations section. Example : We have seen that the final query is : select customer_name, borrower.loan_number, amount from r2,r3 where borrower.loan_number = loan.loan_number which is nothing but a simple join of the relations r2(customer_name, borrower.loan_number), and r3(amount,loan.loan_number), the common attribute being the loan_number. 14

15 15 The consecutive steps are explained in the following figure: Step 1: Step 2: Step 3: 15

16 16 In our algorithm we have tried to minimize the retrieval of large datasets. The cost measure that we have considered is the size of the retrieved result sets. Types of queries dealt with: Although the existing query languages (the SQL) are very powerful, global query optimization typically focuses on a subset of the query language, namely selectproject-join queries with conjunctive predicates. This is an important class of queries for which good optimization opportunities exist. However, there is also an important subset of queries which we did not consider, such as the queries with disjunctions, unions, aggregations, or sorting. Joining of two Relations: If two relations for a join operation are stored in the same site, the join operation is done using the local database server. However, in a number of cases we have to perform join operation of two relations, stored in different sites. In these cases we proceed using semi-join approach. Suppose we need to compute R A S, and size(r) < size(s). We first select S from the first site, then compute Π A (S). Then from R, we select only those rows for which there is one or more match with S on the attribute A, instead of selecting the whole relation R. This operation is known as semi-join, denoted as R S. Then we perform the final join operation as, R A S = (R S) A S. The join operation is done using simple nested-loop join. However, if there is a search operation required, for some predicate followed by the join operation, then we use merge-join instead of simple nested-loop join, because merge-join results in a relation sorted by the values of the attribute of join. 16

17 17 5. Conclusion Here the functionality of JSQL can be stated in a brief. The schemas are stored in different servers, and a middle layer has been implemented to communicate with individual servers. The users are connected to the middle layer, and they perceive the whole distributed system, as if centralized at a single server, and accordingly, they frame their queries. The queries are decomposed, fragments are executed by individual servers, and final results are produced by the middle layer, by combining results from individual servers. The entire software is written using Java SDK 5.0. Here, instead of adopting conventional approach of fragmentation, we store a number of relations in individual database servers. This makes the design of the initial database schema a little difficult. Because, we have to be very cautious with the design, on the point that the functional dependency between the relations hold, else our join operation will no longer remain lose-less, so the DDBMS system will start to produce incorrect results. Also, we did not use replication of data in our system. This saves storage a reasonable extent, and provides a good data isolation, but in the cost of a certain degree of parallelism. Basically, whether we replicate data, or not, should entirely depend on the end-user requirement. Since the last decade, distributed database research has been a very important filed, culminating in the release of a number of first-generation commercial products. If it meets commercial requirements, distributed database technology will impact data processing the same way centralized systems did a decade ago. A well-accepted solution for distributed transaction management and recovery is yet to come up. Microsoft provides an OLE DB based distributed network, which can not utilize the entire degree of concurrency controls, and the system is yet to optimize the performance of distributed query execution. Oracle s distributed system allows a user to do transactions on only one database server at a time, thereby eliminates concurrent processing of transactions. A new DDBMS implemented on the basis of a middle layer, interfaced with existing third party centralized database servers, hence has an ample scope to emerge as a new technology. 17

18 18 6. References 1. M. T. Ozsu, P. Valduriez, Distributed Database Systems: Where are we now? GTE Laboratories, University of INRIA 2. Charles Dye, Oracle Distributed Systems, SPD Publication, M. Stonebraker, Morgan Kaufmann, San Mateo Readings in Database Systems, M. Stonebraker, Future Trends in Database Systems, IEEE Trans. Knowledge and Data Eng., Vol. 1, No. 1, Mar. 1989, pp José A., Blakeley Conor, Cunningham Nigel Ellis, Balaji Rathakrishnan, Distributed/Heterogeneous Query Processing in Microsoft SQL Server, Microsoft Corporation, Redmond 6. Silberschatz, Korth, Sudarshan, Database System Concepts, McGraw Hill 18