HR-SQL 2.0: A Tool for Hypothetical and Recursive SQL on Relational Database Systems

Transcription

1 Error: Macro TOC(None) failed 'NoneType' object has no attribute 'endswith' Error: Macro Image(hrsql_icon.png,5%) failed sequence item 0: expected string, NoneType found HR-SQL 2.0: A Tool for Hypothetical and Recursive SQL on Relational Database Systems 1. Introduction Current relational database management systems (RDBMS's) include SQL as a de-facto query language, following the standard SQL:2011, which evolves from formal query languages as originally proposed for the relational model (Relational Algebra and Relational Calculus). However, as pointed out by many, the relational model has several limitations. In particular, the use of recursion is severaly limited and mutual recursion is not allowed. Here, we include a tool (former version 1.0 can be found at HR-SQL 1.0), for the database language and system HR-SQL, addressing two main lines for enhancing SQL: broadening recursion and incorporating hypothetical relation definitions and queries. Moreover, we also support aggregates in this context. This tool version fixes the former, adds several commands, includes an integrated development environment (ACIDE) that in particular allows to graphically display the dependency graph, supports relation and column aliases, an ODBC bridge to external DBMS's,... This new version of the tool has been developed from scratch with materials from DES. The tool generates a Python script for the DBMS of choice (DB2, MySQL, Oracle, PostgreSQL, and SQL Anywhere) that, upon execution, materializes in the DBMS the relations defined in a file. Afterwards, such relations can be queried either with the native SQL or with the extended SQL as defined in HR-SQL. Further versions will deal with other RDBMS's as Access and SQL Server. 2. Brief Description The basic syntax of the language is depicted in the next figure: Error: Macro Image(HRSQLGrammar.2.png,47%) failed sequence item 0: expected string, NoneType found In addition, the system enhances this basic syntax by adding: Both relation and column aliases. Grouping by a list of expressions. Including the ORDER BY clause.... An example of a database definition for the recursive generation of numbers from 0 up to 100 follows: HR-SQL 2.0: A Tool for Hypothetical and Recursive SQL on Relational Database Systems 1

2 numbers(n integer) := SELECT 0 UNION SELECT n+1 FROM numbers WHERE n<100; First, note that, by contrast to standard SQL, we only refer to relations (instead to tables and views as in standard SQL), which are specified by a prototype and an SQL statement that defines the relation. The prototype is relname(typed_arguments) with the relation name and the sequence of typed argument names between parentheses, similar to what follows to an SQL CREATE TABLE statement. Second, note also that from-less SQL statements are supported, as SELECT 0;, which can be understood as SELECT 0 FROM dual;, where dual is the one-row, one-column void table as in Oracle DBMS. Finally, the recursive relation is specified in a more compact way if compared to standard SQL: CREATE RECURSIVE VIEW numbers(n) AS WITH n_cte(n) AS SELECT 0 FROM dual UNION ALL SELECT n+1 FROM n_cte WHERE n<100 SELECT * FROM n_cte; An example of a database definition for the recursive, non-linear Fibonacci example follows: fib(n integer, f integer) := SELECT 0,1 UNION SELECT 1,1 UNION SELECT fib1.n+1, fib1.f+fib2.f FROM fib AS fib1, fib AS fib2 WHERE fib1.n=fib2.n+1 AND fib1.n<10; Here, we make use of the relation name aliases fib1 and fib2. An example of a database definition for mutually-recursive generation of even and odd numbers follows: even(x integer) := SELECT 0; odd(x integer) := SELECT even.x+1 FROM even WHERE even.x<100; even(x integer) := SELECT 0 UNION SELECT odd.x+1 FROM odd WHERE odd.x<100; This defines a couple of relations for specifying even and odd numbers from 0 up to 100 with a mutual recursive definition. Here, a void definition for even is given in the first line in order for the parser to have its prototype before the odd relation that uses it. This void definition is replaced in the third line by the proper one. More details for the previous tool can be found at: G. Aranda-López, S. Nieva, F. Sáenz-Pérez, and J. Sánchez-Hernández, "Incorporating Hypothetical Views and Extended Recursion into SQL Database Systems", EPiC Vol. 26, July, An ongoing paper which develops on HR-SQL 2.0 is under writting. 2. Brief Description 2

3 3. Downloading and Installing Prerequisites ODBC installed with already-defined DBMS connections. Windows and MacOS include ODBC by default, while Ubuntu does not. In this case, unixodbc can be installed. Make sure to modify des.ini as warned in Section Installing and Executing later on. Python 3.3 with pyodbc for this version. See also specific requisites for each distribution. Available Distros HR-SQL (Current Stable Version) 1. Executable for Windows OS's and Java IDE HRSQL Windows.zip. This needs Java 1.7 or higher for starting the IDE, but if only the binaries are used, there is no need. 2. Sources for SWI-Prolog HRSQL SWI.zip. This needs SWI-Prolog installed. 3. Sources for SICStus Prolog HRSQL SICS.zip. This needs SICStus Prolog installed. This release has been tested on Windows 7 with all the included examples for the following DBMS's: DB MySQL PostgreSQL SQL Anywhere 16 and the following Prolog systems: SWI-Prolog SICStus Prolog But it is expected that Oracle, Access and SQL Server also fit the bill. The file hrsql_hrsql.sql in sources contains the HR-SQL system. Other files come from DES and have been used to reduce development time. More versions for other platforms are expected in the near future. ==== Release Notes for HR-SQL 2.0.1==== Stratification algorithm follows [ANSS13] instead of the former finite domain implementation. Non-delimited identifiers starting with $ were rejected in PostgreSQL. Introducing a remark at the end of the file resulted in an exception. Some aggregate expressions received incorrect inferred types. SWI-Prolog supported (last version supported Prolog 6.6.6). More examples have been included. Changes: By default, an ODBC connection name postgresql instead of mysql is expected. See later to know how to change the default ODBC connection. 3. Downloading and Installing 3

4 Known bugs: Unknown data types for SICStus Prolog ODBC library raise an exception. SWI-Prolog ODBC library simply returns the value as an atom (even if it is a numeric type). This situation appears when submitting straight SQL queries. Caveats: SQL Anywhere does not permit the column name time which are used in the examples. A few tests do not pass for MySQL due to some limitations in its dialect which may be overcome in the future. HR-SQL Executable for Windows OS's and Java IDE HRSQL.2.0.Windows.zip. This needs Java 1.7 or higher for starting the IDE, but if only the binaries are used, there is no need. 2. Sources for SWI-Prolog HRSQL.2.0.SWI.zip. This needs SWI-Prolog installed. 3. Sources for SICStus Prolog HRSQL.2.0.SICS.zip. This needs SICStus Prolog installed. Installing and Executing Installation is pretty straightforward: Both the executable and source versions simply need to decompress the given zip file into any folder. Warning: The system is preconfigured to start with the ODBC connection name postgresql to a PostgreSQL database. If the connection does not exist, then an error is raised and the system must be restarted with an appropriate setting. If you have another connection name and/or database system (including DB2, MySQL, Oracle, PostgreSQL, and SQL Anywhere) you have to edit the file des.ini, locate the third line including /silent /hrsql postgresql and change postgresql to your ODBC connection name of choice. Executable Version You can either execute the plain binaries or the Java-based IDE (which in turn calls to the console application hrsql.exe). Plain binaries: hrsql.exe A simple console application. Type hrsql at an OS console (cmd) to execute Error: Macro Image(HRSQLDOS.png,45%) failed sequence item 0: expected string, NoneType found hrsqlwin.exe A windows console application. Type hrsqlwin at an OS console (cmd) to execute Error: Macro Image(HRSQLWIN.png,45%) failed sequence item 0: expected string, NoneType found Java IDE: hrsql.java. Two options are available: HR-SQL (Current Stable Version) 4

5 Simply double click this file in Windows explorer Call this from an OS console (cmd) with java -jar hrsql.java Error: Macro Image(HRSQLIDE.png,45%) failed sequence item 0: expected string, NoneType found Source Version First, you need either SWI-Prolog or SICStus Prolog and the corresponding HR-SQL zip file. Then, execute the Prolog interpreter, change the working directory to the folder in which HR-SQL sources are located (see working_directory/2 for SWI-Prolog and current_directory/2 for SICStus Prolog), and load the HR-SQL system with:?- [hrsql].... ********************************************************* * * * HR-SQL 2.0.1: A Hypothetical and Recursive * * Framework for SQL Databases * * * * Susana Nieva (1) * * Fernando Saenz-Perez (2) * * Jaime Sanchez-Hernandez (1) * * UCM GPD (1)DSIC (2)DISIA * * * * This program comes with ABSOLUTELY NO WARRANTY, is * * free software, and you are welcome to redistribute it * * under certain conditions. Type "/license" for details * ********************************************************* - HR-SQL Commands /process_db <db_file> : Load and process a database /load_db <db_file> : Load a database /process_db : Process the loaded database /transform_db : Transform HR-SQL to R-SQL DB /list_relations : List the relations /pdg : Display the dependency graph /strata : Display the strata /help : Show this help /quit : End system session Other inputs are processed as SQL queries HR-SQL(MySQL)> 4. System Description The Java-based IDE (see previous figure) includes a number of features for easing the playing with the HR-SQL system, such as syntax highlighting, project management (left upper pane), multiple editor windows (middle upper pane), relocatable panels, graphical display of the dependency graph (right upper pane labelled with PDG) after the transform stage (where N1 -> N2 means that N2 depends on N1, and negative dependencies are shown in red). A toolbar includes shortcuts to the most common commands which are described in the next section. For example, the button "process db Editor" submits the command "/process db" for the database as displayed in the active editor window. The button "process db File" submits the same Executable Version 5

6 command but a dialog pops up requesting the file on which the command is to be applied. Finally, the button "process db" submits the same command referring to the already-loaded database. The most representative commands are described next. 4.1 Commands /hrsql connection Select the given ODBC connection as the default bridge to the external RDBMS when generating the Python script. /process_db file Load and process the HR-SQL database defined in the given file. This is the all-in-one command that completely processes an HR-SQL file giving as a result the materialized user-defined relations in the external database. This command applies all the stages: Preprocessing, which removes nested ASSUME?s; the iterative application of the function tr, which replaces a single assumption and results in an R-SQL database; the generation of the Python script for the selected database; the execution of this script (resulting in the materialization); and the display of all the tuples for each user-defined relation. /load_db file Load the HR-SQL database defined in the given file. This is useful for inspecting the result of the preprocessing stage. /process_db Once a database is loaded (either an HR-SQL or R-SQL), it can be processed with this command. This command applies the transform stage on. /transform_db Transform the loaded HR-SQL database into an R-SQL database (with the iterative application of tr). This command is useful after /process_db for inspecting the result of the transform stage. /list_relations List the relations for the current stage. /pdg Display the dependency graph, first the list of nodes (relation names with their arities: Relation=Arity), and then the arcs N1+N2 (N1-N2, resp.) indicating that node N1 (negatively, resp.) depends on node N2. This graph is represented in the ACIDE panel PDG by pressing the Refresh button. /strata Display the stratification as a list of pairs (N,S), where N is a node and S is its stratum number. As an optional parameter, it can include the name of a relation, and then, it displays the dependency graph restricted to the nodes which are reachable from that relation. Other useful OS-oriented commands are: /pwd, which displays the current directory. /cd path, which sets the current directory to path. /ls, which displays the contents of the current directory. /log file, which enables output to be logged in file. Use /nolog to stop logging. The implementation of HR-SQL is based on stratified fixpoint techniques, similar to the Datalog language. But in order to compute the meaning of a database (the tuples defined by the query of each relation), first of all, the ASSUME clauses must be eliminated from the source database dealing to an R-SQL database. This is a non-trivial procedure performed in two stages (where both introduce auxiliary relations): A preprocessing stage that extracts nested ASSUME clauses, obtaining a non-nested form for hypothetical queries. The second stage, transform, eliminates ASSUME?s on successive iterations of a transformation function tr. This function 4. System Description 6

7 is applied until no more ASSUME clauses are found. After applying this transformation, a stratification is computed for the relations of the database (if exists), in order to define the order in which relations are computed. The idea, as in Datalog, is to collect mutually recursive relations in the same strata to be computed simultaneously. In the case that the definition of a relation r involves another relation s in a negative form, then s must be completely evaluated before r, i.e., s must be in a strata less then the one for r. The notion of being involved in a negative form is a bit elaborated in HR-SQL as it depends not only on the explicit negation (NOT), but also on ASSUME clauses and aggregates. Moreover, HR-SQL still uses additional criteria for scattering relations over strata for efficiency reasons. The stratification is then used to arrange the computation of the fixpoint for the relations of the database. 5. System Session Here, we process the first recursive example: HR-SQL> /process_db C:\HR-SQL 2.0.1\examples\numbers.sql HR-SQL> numbers(n integer) := SELECT 0 UNION SELECT n+1 FROM numbers WHERE n<100; Application of preprocessing gives: numbers(n INTEGER) := ( SELECT 0 ) UNION ( SELECT n + 1 FROM numbers WHERE n < 100 ); R-SQL Database: numbers(n INTEGER) := ( SELECT 0 ) UNION ( SELECT n + 1 FROM numbers WHERE n < 100 ); Original Strata : [(numbers/1,1)] Extended Strata : [(numbers/1,1)] Unfiltered Strata : [(numbers/1,1)] Final Strata : [(numbers/1,1)] Python Script for R-SQL Database: ############################################################### # HR-SQL # Python script for: # - Database Filename : C:\HR-SQL 2.0.1\examples\numbers.sql # - This Python Filename: script.py # - ODBC Connection : mysql # - RDBMS : MySQL # - Prolog System : sicstus # - Date : # - Time : 23:20:29 # # Run this script with the ODBC connection defined already at # the OS level and the database system up and running. ############################################################### # Importing modules: import pyodbc import sys # Try to open the ODBC connection: try: conn = pyodbc.connect("dsn=mysql") except: 4.1 Commands 7

8 print("error opening connection 'mysql'.") sys.exit() # Cursor definition: cursor=conn.cursor() # Creating RN\RN' Tables: try: cursor.execute("drop TABLE numbers;") except: pass cursor.execute("create TABLE numbers(n INTEGER);") # Creating RN' Temporary Tables: # Stratum 1: cursor.execute("insert INTO numbers(n) SELECT ALL * FROM (SELECT ALL 0 FROM dual) AS alias2") # Views for DBMS without EXCEPT: try: cursor.execute("drop VIEW numbers_temp;") except: pass cursor.execute("create VIEW numbers_temp(n) AS SELECT ALL n + 1 FROM numbers WHERE n < 100") ch = True while ch: newtuples = 0 cursor.execute("insert INTO numbers(n) SELECT ALL * FROM (SELECT ALL * FROM numbers_temp WHERE (n) NOT IN ( SELECT ALL * FROM numbers AS alias0 )) AS alias1") newtuples = newtuples + cursor.rowcount if (newtuples == 0): ch = False # Dropping Temp Views: cursor.execute("drop VIEW numbers_temp;") # Dropping RN' Temporary Tables: # Commit changes: conn.commit() # If successful, print Success: print("success.") Success. Info: Relation numbers: { answer(0), answer(1),... answer(100) } Info: 101 tuples computed. Next, we submit a hypothetical query to this database: FROM numbers WHERE n BETWEEN 50 AND 100 IN numbers SELECT * FROM numbers; ( SELECT 0 ) UNION ( SELECT n + 1 FROM numbers_0 WHERE n < 100 ) ) UNION ( SELECT 2 * n FROM numbers_0 ); ( SELECT 0 ) UNION ( SELECT n + 1 FROM numbers_1 WHERE n < 100 ) ) UNION ( SELECT 2 * n FROM numbers_1 WHERE ELECT 0 ) UNION ( SELECT n + 1 FROM numbers WHERE n < 100 ); ( SELECT 0 ) UNION ( SELECT n + 1 FROM numbers_2 WHERE n < 100 ) ) UNION ( SELECT 2 * n FROM numbers_2 WHERE CT * FROM numbers_2; 5. System Session 8

9 bers_2/1,1),(answer/1,2),(numbers/1,3),(numbers_0/1,4),(numbers_1/1,5)] atabase: ###################################### query at the system prompt. e: script.py : mysql : MySQL : sicstus : : 23:46:42 e ODBC connection defined already at database system up and running. ###################################### nnection: DSN=mysql") nnection 'mysql'.") BLE answer;") BLE answer(n INTEGER);") Tables: BLE numbers_2(n INTEGER);") TO numbers_2(n) SELECT ALL * FROM (SELECT ALL 0 FROM dual) AS alias3") EXCEPT: EW numbers_2_temp;") EW numbers_2_temp(n) AS SELECT ALL n + 1 FROM numbers_2 WHERE n < 100 UNION SELECT ALL 2 * n FROM numbers_2 W INTO numbers_2(n) SELECT ALL * FROM (SELECT ALL * FROM numbers_2_temp WHERE (n) NOT IN ( SELECT ALL * FROM num cursor.rowcount = False 5. System Session 9

10 numbers_2_temp;") TO answer(n) SELECT ALL * FROM (SELECT ALL * FROM numbers_2) AS alias2") Tables: E numbers_2;") ccess:. You can also submit native SQL queries which do not need to be handled by HR-SQL but otherwise passed straight to the underlying RDBMS (MySQL in this case). HR-SQL> select * from numbers; answer(n:integer(4)) -> { answer(0), answer(1),... answer(100) } Info: 101 tuples computed. Contact This sofware has been developed by Susana Nieva (nieva at sip.ucm.es), Fernando Sáenz Perez (fernan at sip.ucm.es) and Jaime Sánchez Hernández (jaime at sip.ucm.es), at the Declarative Programming Group of the Universidad Complutense de Madrid, Spain. Contact 10