Web Services for Relational Data Access

Transcription

1 Web Services for Relational Data Access Sal Valente CS 6750 Fall 2010 Abstract I describe services which make it easy for users of a grid system to share data from an RDBMS. The producer runs a web services based server, and the consumer runs a web services based client. Authentication, naming, and routing issues are all handled by the grid infrastructure. The producer has fine-grained control over which data sets to share. The server always provides fresh data from the DBMS. 1 Introduction 1.1 Background Grid computing is based on software that provides secure and managed sharing of networked computational resources. Typically, each shared resource is backed by a server. For example, a file is a shared resource. I can store data in a file by sending a write request to the server, and then other users can access that data by sending a read request to the same server. A CPU is also a shared resource. I can share my CPU by running a server, so that other users can execute jobs on my CPU by sending an execute job request to the server. Genesis II is grid middleware software which consists of servers and clients for sharing several types of resources, including file data, job queues, naming services, and authentication services. A grid may contain millions of files and many thousands of other types of resources, so the software must provide a userfriendly abstraction for finding and managing all of these. Genesis II abstracts all resources into the file system. If you are using a Genesis II client and you want to access a resource which is hosted on a Genesis II server, then you must specify the pathname of the resource, not the URL or any other type of network identification. This makes it relatively easy to find and access any resource from anywhere. The idea of abstracting non-file objects into the file system is used in the Unix /dev file system. Each resource has access control lists for read, write, and execute access. All Genesis II resource types are defined as web services. Web services are defined in a way that is machine-independent and language-independent. To define a web service, you define a set of operations, where each operation 1

2 has a name, an input message, and an output message. The input and output messages are defined in terms of generic types integers, strings, arrays, and so on. For example, the file resource type has (at least) two operations, named read and write. The input to read is a pair of integers the file offset and the block size. The output from read is an array of bytes. Once a web service has been defined, both clients and servers can be written (or partially automatically generated) for that service in a variety of computer languages for a variety of computer systems. Thus, any new web services that are defined for the Genesis II system may be usable in other systems as well. 1.2 Motivation Say that a group of researchers are using the grid for computational science. Every day, they get a new set of input data, they analyze the data, and they write the results into a relational database. The researchers want to share the results not just among the members of the group, but with other researchers around the world. However, they don t want to configure their DBMS with the other researchers security credentials, firewall configuration, etc. They want to share the data through the grid infrastructure, as defined in the previous section. Sharing relational data should be as easy as sharing file data. 1.3 Solution I have designed a pair of resource types which facilitate sharing of relational data through a grid. A resource of the DBConnection type represents a JDBC connection to an external DBMS system. A resource of the DBView type represents a specific view a specific set of tables, columns, and rows. When a user sends a read request to this resource, the server executes the SQL statement and returns the data as plain text in CSV format. For example, say that my sales database contains a products table and an orders table. I can create a resource which represents joining these two tables and selecting the rows where the customerid is some specific person. Then I can add that person s grid account to the resource s ACL. Now, that person can check his current list of orders by reading from a named resource using the exact same command as he would use to read a standard text file in the distributed file system. I did not need to give him direct access to my DBMS, or even tell him where the data is stored. I have implemented clients and servers to support these new resource types in Genesis II. This paper is organized as follows: Section 2 describes the DBConnection resource type. Section 3 describes the DBView resource type. Section 4 describes the operations that are defined for the DBView resource type. Section 5 expands the design to support views made up of tables from multiple DBMS systems. Section 6 discusses related work and section 7 concludes. 2

3 2 Database Connection Type The first new resource type is called DBConnection. It represents a JDBC connection to an embedded or external DBMS. When you create a resource of this type, you specify the database connection parameters: the JDBC URL and the database login name and password. This resource is effectively a proxy it communicates with a database as a database user on behalf of a grid user. A grid user with read access to this resource will be able to read all of the tables that are accessible by the specified database user. There is only one operation defined for this resource type. The operation is named createview. It creates a new DBView resource which can be used to read from a single table in the DBMS. There is no operation which would allow a user to insert or modify data in the database. From a security perspective, you should be aware that a grid administrator (with the access to look inside the internal state of local resources) may be able to extract the login and password parameters from a DBConnection resource in order to gain write access to the database. 3 Database View Type The other new resource type is called DBView. It represents a single table, view, or select statement in a DBMS. To create a new View resource, you send a createview request to a Connection resource. Internally, the View resource stores a connection-resource-key, which is a piece of state data that allows the View to communicate with the Connection. Each time that the View needs to read from the database, it uses this resource key. Thus, if you change the username or password in a Connection resource, then your change will affect all existing Views that were created through that resource. If you delete a Connection resource, then all of its Views will immediately stop working. The internal state of a View resource also contains a list of tables and a where clause written in SQL. For each table, the state contains a list of columns to select. For each column, the state can contain an alias, also called a header for that column in the output. A View resource is always created as a simple complete view of a single table. The View resource type defines four operations for modifying views: select, project, rename, and join. The design of these operations is based on relational algebra. Using these operations, a user can create arbitrarily complex views. I designed this resource type to be used by three different sets of users: 1. The administrators. They create the Connection resource and the set of single table View resources. In the example from the section 1 of this paper, an administrator would create views for the products table and the orders table, and he would make these views readable by the second set of users. 3

4 2. The producers. They combine the simple views into complex views. To continue the example from section 1, a producer would create a view that joins the products and orders tables and selects the appropriate rows. He would make this view readable by the third set of users. 3. The consumers. They only read data from existing views. In order to support views that are produced and consumed by different users, the design adheres to the following rule: You do not need to know how a view was created in order to access and modify that view. For example, say that I create a view with the clause age > 40. When I give you this view, you see what appears to be a simple table where all of the rows happen to have age > 40. You don t know whether the database contains any other rows, or whether the view contains a where clause. You are free to rename the age column, or to do a projection which removes the age column, or to do a selection which further restricts the set of rows. There is nothing that you could do that would break the view or cause it to show data where age < 40. The next section discusses the design and implementation of the View resource type operations. 4 Database View Operations 4.1 Select The select operation restricts the rows that are included in the view. After a select operation, a view will display a subset of the rows that it displayed before the operation. The select operation takes a single boolean expression. The expression is written in a language which is a subset of SQL. The expression may contain column names, boolean operators, and mathematical operators. Also, it may contain SQL case statements and SQL between statements, and it may contain the SQL like operator and the SQL in (x,y,z) operator. The expression may not contain nested select statements or calls to SQL functions. We can refer to this subset of SQL as the select operation language. Here is an example of a valid expression written in this language: (Column1 + Column2 < Column3) OR (Column4 like x% ) I wrote an antlr grammar file for the select operation language. Then I used antlr to build a parser for the language. When you send a select request, the server parses the expression into a parse tree, and then it unparses the tree back into an expression which may or may not be identical to the original expression. This serves two purposes: It verifies that the expression is written in the acceptable subset of SQL. It defends against SQL injection attacks. 4

5 The column names in the expression must be translated. The user specifies a column by the header that he observes in the view, but when the resource sends the select statement to the DBMS, it must specify column names by the actual names of tables and columns. Using a parse tree makes it easy to find and translate the column names. After the select operation parses and unparses the expression, it updates the resource s internal state. This is easy. If the resource does not yet have a where clause, then we save the unparsed expression as the resource s where clause. Otherwise, we take the old clause, and we append the SQL keyword AND, and we append the new expression in parentheses. There is no need to combine the old and new clauses in any complex way. The view is restricted to the rows that meet both the old restrictions and the new restrictions. 4.2 Project The project operation restricts the columns that are included in the view. After a project operation, a view will display the same number of rows as it displayed before the operation, but with a subset of the columns. The project operation takes a list of column headers. The operation modifies the resource s internal state. It loops over each table in the view, and for each table, it loops over each column. For each column, if that column s header is not among the names in the specified list of headers, then that column is removed from the view. This operation does not affect the resource s where clause. It is perfectly legal to continue to restrict the view s rows based on a column which is no longer displayed. However, after a column has been projected out of a view, there is no way to add new clauses based on that column. 4.3 Rename The rename operation changes a column header in the view. The column to rename is specified by its current header. The new column header must be unique you may not rename a column to X if the view already has a column with a header of X. This operation does not affect the resource s where clause. Internally, there where clause is always stored in terms of the actual column names in the database, not the headers that are displayed in the view. Recall that the select operation translates from column headers to database column names. 4.4 Join The join operation modifies the view by doing a natural join with another view. The user sends a join request to a resource. Let s call this resource A. The request is to modify resource A by joining it with a specified resource. Let s call this resource B. 5

6 First, the server reads resource B s internal state data. Recall that a view s internal state contains a list of tables. Each table is added to resource A s list of tables. Then, the server looks for any column header that appears in resource A s old tables and also appears in the new tables that were copied from resource B. Recall from the description of the rename operation that each column header in a view is unique. If a column header appears in one of the tables, then it may not appear in any other table. We must ensure that each column header in the joined view is unique. When we find a header that is in both an old table and a new table, we do a natural join of those columns. We remove the column from the list of columns displayed in the new table. Then, we add to the resource s where clause. The new clause is oldtable.column = newtable.column. Note that it doesn t matter whether we hide the column in the old table or in the new table. As a result of the new where clause, the data displayed in the old column is identical to the data that would be displayed in the new column. Finally, the server adds the where clause from resource B to resource A. This is as easy as adding a new clause in the select operation. The joined where clause is simply the old clause and the new clause. After the join operation completes, resource B is irrelevant. That resource may be modified or deleted without affecting resource A. What if there is a table which appears both in resource A s list of tables and in resource B s list of tables? It s not a problem. A single table may appear multiple times in a view s list of tables. Consider any column in that table. If the column has not been renamed in either of the instances, then that column will be the target of a natural join, so one instance of the column will be hidden. However, if the column was renamed in one of the instances, then the view can contain both instances of that column. The two columns will have different headers, and they may contain different data, depending on how the tables were joined. Say that resource A contains some table named X, and that resource B also contains some table named X. As just explained, the joined view may use columns from both of these tables. Therefore, before these resources can be joined, one of the tables must be aliased. The table alias is not visible to the end-user, so the server generates a random unique alias. It stores the alias in the resource s internal state, so that the resource will execute select from table alias. Then, the server parses the resource s where clause. Each instance of table.column is changed to alias.column. The server aliases all necessary tables in one of the resources, and then it joins the resources. What if we want to do a join which is not a natural join? We simply use the rename operation in order to make it a natural join. For example, say that resource A has columns named X and Y, and that resource B has columns named Y and Z. Say that we want to join these views on X = Z, but we do not want to join on Y = Y. First, we can optionally make a clone of resource B, in case we want to continue to have access to a view with columns named Y and Z. Then, in resource B (or its clone), we rename column Z to X, so that it will be a target of the natural join. Finally, we rename column Y to Y2, so that it 6

7 will not be joined. 4.5 Open Stream The DBView resource type is a superclass of the Streamable ByteIO Factory resource type. When any Genesis II client wants to read data from any resource of this type, the client sends an open stream request, and then it sends a series of read requests, and finally it sends a request to destroy the stream. In response to the open stream request, the resource uses its connectionresource-key to get the JDBC parameters from the local database. (The local database should not be confused with the external database that is referenced by the JDBC parameters). Then, the resource generates the SQL select statement with the table names, aliases, column names, headers, and where clause from its internal state. The resource opens a JDBC connection and sends the SQL statement. It reads the resulting table from the JDBC connection, and it writes the data to a local temporary file. After it has copied the entire table, it closes the JDBC connection and responds to the open request. In response to the read requests, the resource reads data from the local file and sends it to the client. In response to the destroy request, the resource deletes the local file. I believe that it would be more efficient if the resource sent the data directly from the JDBC connection to the Genesis II client. This process would avoid the possibly long and costly step of reading and writing the entire table to a file. However, it would make the code significantly more complex, and I question whether the gains would be non-negligible. 5 Multiple Database Views The previous section described the design of five operations select, project, rename, join, and open stream all based on the assumption that a view is always confined to a single DBMS. In this section, we drop that assumption. The goal is simple. Say that resource A is a view with columns X and Y from some database, and resource B is a view with columns Y and Z from some other database. The user should not need to know or care that the data comes from different databases. It should be possible to join these two views. When we read from the joined view, we should see a table with columns X, Y, and Z, joined on the rows where Y = Y. Everything should just work as if the data came from a single database. 5.1 MultiDB Internal State Recall that the internal state of a DBView resource consists of a connection resource key, a list of table parameters, and a where clause. Let s call a complete instance of this state an object of the DBSelectParams class. Given 7

8 a DBSelectParams object, we can generate a complete SQL statement, and we can read a complete data set. In order for a DBView resource to represent data from multiple DBMS systems, the internal state of the resource must contain a list of DBSelectParams objects. The next subsections describe how we create and use this state. 5.2 MultiDB Join The user sends a join request to resource A to join it with resource B. The server reads resource B s internal state data, and finds that it does not use the same connection-resource-key as resource A. In this case, the server simply appends a copy of resource B s DBSelectParams object to the end of resource A s list of parameters. The operation does not attempt to combine the DBSelectParams in any way. It does not look for matching table names or matching column names. It does not combine the where clauses. It just makes a list. If the user joins a third view to resource A, and the third view uses yet another connection-resource-key, then the operation simply adds a third parameters object to the resource s internal state. If resource A contains a list of n DBSelectParams objects, and resource B contains a list of m DBSelectParams objects, then joining them is a two-step process. The first step is to concatenate them into a list of n + m objects. The second step is to find any pair of objects that have the same connection-resourcekey. Those two objects must be combined using the join operation from section Open Stream Here is how the open stream operation works if the resource contains multiple DBSelectParams objects. The server uses the first object to build an SQL statement and read the resulting data set into memory. Then, it uses the second object to read a second data set into memory. Then, it joins the two data sets into a single results table. The next subsection describes the function that I wrote to join memory-resident data sets. If the resource contains three or more DBSelectParams objects, then the server joins the results table with the third table, and so on. In the end, we have a single results table. The server writes this table to a local file in order to support the read requests that always follow an open stream request. The way that the server processes read requests is unchanged. 5.4 Internal Join Function I implemented a function that joins two data sets in memory. Each data set is stored as an array of Row objects, where a Row object is an array of strings. To join the data sets, the server finds the columns in the second data set that have headers that also appear in the first data set. These are the key columns. 8

9 Using the list of key columns, the server can easily convert from a Row to a Key, which is an array containing the strings from the row s key columns. The function creates a Map data structure where an entry is a mapping from a Key object to a dynamic list of Row objects. The function processes the second table: For each row in the table, lookup the key in the map. If the key is found, then add the row to the entry s list of rows. Otherwise, add a new entry mapping the key to a new list with the row. After the server organizes the second table as a map, it processes the first table: For each row in the table, lookup the key in the map. Find a list of n rows, where n may be zero. Combine the current row with each of those n rows. Add the new n rows to the results table. That s a simple join function. There may be algorithms that are more efficient or more tunable for certain types of data sets, but the function described here is sufficient. 5.5 MultiDB Select The select operation parses the given boolean expression. It finds all of the referenced column headers. If all of the columns appear only in a single DBSelectParams object, then it updates that object as described in section 4. If the columns are spread out across multiple databases, then it is not safe to update any individual DBSelectParams object. For example, say that the new clause is X=1 OR Z=2 where X and Z come from different databases. In this case, we cannot select just the rows where X = 1 in the first database, and we cannot select just the rows where Z = 2 in the second database. So we cannot update either of those sets of parameters. We must record the new clause somewhere else in the view s internal state. In the open stream operation, after we generate the results table from all of the DBSelectParams objects, the final step is to apply the view s global where clause. I have not implemented this feature. The current implementation rejects select operations where the columns are spread across multiple databases. I consider this an area for further work. 5.6 MultiDB Project and Rename The project and rename operations are basically unchanged. The project operation simply projects the new set of headers in each individual DBSelectParams object. The rename operation simply renames the column in each individual DBSelectParams object. 6 Related Work There are several existing services for sharing relational data among grid users. Of these, I have been influenced by the WS-DAIR specification. This is the 9

10 relational data specification from the WS-DAI family of specifications [1]. The authors of these specifications compared their goals to the goals of Embedded SQL and JDBC: The development of standards for accessing databases from programming languages has saved countless hours of developer effort through improved portability and interoperability. The WS-DAI family of specifications should bring similar benefits to service-based computing. My project is focused more on benefits to the end user rather then to the developer. A more ambitious related project irods (Rule-Oriented Data management System) [2]. irods is grid middleware software that implements a virtual storage system for research data. My project is also a kind of virtual storage system backed by one or more physical storage system, but I aim for a smaller and easier interface. The specific set of DBView operations were influenced by relational algebra [3]. 7 Conclusion I have designed a pair of resource types for sharing relational data in a web services based distributed system. These resource types allow a provider to carefully select what data he wants to share, and with whom. The design makes it easy for a consumer to download a fresh instance of the data in a convenient format. I have implemented my design in the Genesis II system. The source code is open-source. It includes: WSDL definitions of the two resource types an implementation of the resource operations in Java an internal join function an antlr grammar file a command-line client using the Genesis II command tool library There is significant opportunity for future work. First and foremost, I have not yet verified that my work is as useful and convenient as I believe it to be. A survey should be done of real scientific researchers. The DBView type can be expanded to include other relational algebra operations such as outer joins, unions and intersections, selection of distinct rows, sorting, and aggregate functions such as row groupings. The MultiDB support is incomplete and should be finished. Finally, the system should be expanded to support the joining of views where the resources themselves (as opposed to the databases that they represent) are hosted on different servers. 10

11 References [1] M. Antonioletti, A. Krause, N. W. Paton, A. Eisenberg, S. Laws, S. Malaika, J. Melton, and D. Pearson. The WS-DAI family of specifications for web service data access and integration. SIGMOD Rec., 35:48 55, March [2] M. Hedges, A. Hasan, and T. Blanke. Management and preservation of research data with irods. In Proceedings of the ACM first workshop on CyberInfrastructure: information management in escience, CIMS 07, pages 17 22, New York, NY, USA, ACM. [3] R. Ramakrishnan and J. Gehrke. Database Management Systems. McGraw Hill, Third edition,