DENODO VIRTUAL DATAPORT 4.6 ADMINISTRATION GUIDE

Transcription

1 DENODO VIRTUAL DATAPORT 4.6 ADMINISTRATION GUIDE Update Aug 16 th, 2011

2 NOTE This document is confidential and is the property of Denodo Technologies (hereinafter Denodo). No part of the document may be copied, photographed, transmitted electronically, stored in a document management system or reproduced by any other means without prior written permission from Denodo. Copyright 2011 This document may not be reproduced in total or in part without written permission from Denodo Technologies.

3 INDEX PREFACE...I SCOPE...I WHO SHOULD USE THIS DOCUMENT...I SUMMARY OF CONTENTS...I 1 INTRODUCTION GENERAL ARCHITECTURE PHYSICAL LAYER LOGICAL LAYER Data Module: Cache USER LAYER INSTALLATION AND INITIAL CONFIGURATION EXECUTION EXECUTING THE VIRTUAL DATAPORT GRAPHIC ADMINISTRATION TOOL DOCUMENTATION AND ONLINE HELP MAIN AREAS OF THE ADMINISTRATION TOOL TOOL PREFERENCES Locale Connection Parameters INVALIDATE CACHE SEARCH Quick Search Catalog Search QUERY MONITOR VISUAL CREATION OF VIEWS ACCESS TO THE GRAPHIC ADMINISTRATION TOOL CREATING AND MANAGING PROJECTS Moving Elements to Other Projects IMPORTING DATA SOURCES AND CREATING BASE VIEWS Importing JDBC-Type Sources Deleting a Data Source Creating Base Views from a JDBC Data Source Viewing the Schema of a Base View Importing Web-Service-Type Sources Creating Base Views from a Web Service VISUAL CREATION OF VIEWS AND QUERIES Creating Union-Type Views Creating Join-Type Views Creating Projection / Selection-Type Views Creating Aggregation-type Views Flattening of Register-array-type Fields Viewing the Schema of a View Tree View Mode Data Lineage... 47

4 5.4.9 Executing Views Using VQL Shell Editing / Replacement of a View Definition Source Refresh Views Affected by Modifications OTHER TYPES OF SOURCES Path Types in Virtual DataPort ODBC Sources Multidimensional Database Sources XML Sources JSON Data Sources Web Sources (WWW) Delimited File Sources Google Enterprise Search / Mini Sources Aracne Sources LDAP Sources BAPI Sources CUSTOM Sources JDBC/ODBC sources through SQL Query Data Source Configuration Properties ADVANCED CONFIGURATION OF VIEWS Internationalization Configuration Configuring the Cache of a View Swapping Configuration Editing the Execution Plan Query Capabilities View Configuration Properties Configuration Properties for Specific View Types STORED PROCEDURES Importing Stored Procedures Executing Stored Procedures Use of Stored Procedures in Creating Views PUBLICATION OF WEB SERVICES WEB SERVICE TYPES PUBLISHING WEB SERVICES Operations for Updatable Views Configuration of Connections in the Web Services Published SOAP XSLT Transformations REST XSLT Transformations SOAP over JMS Mappings in the Publication as RSS Web Services Authentication Types Conversion Table for REST / SOAP Published Web Services INVOKING THE EXPORTED WEB SERVICES HTML Output Configuration JSON Output Configuration PUBLICATION OF VIEWS AS WIDGETS PUBLISH A VIEW AS A WIDGET AUXILIARY WEB SERVICES EXPORT TO JSR-168 OR JSR-286 PORTLET EXPORT TO OPENAJAX WIDGET DEPLOYMENT OF AN OPENAJAX WIDGET

5 7.6 EXPORT AS MICROSOFT WEB PART DEPLOYMENT OF A MICROSOFT WEB PART SERVER ADMINISTRATION CONFIGURING THE SERVER Server Connectivity Threads Pool Configuring the Cache Limiting the Number of Concurrent Requests Default Configuration of HTTP Proxy Configuring the Wwapping to Disk Parameters Configuring Runtime Parameters for Stored Procedures Configuring Default Internationalization JMS Listeners EXPORTING / IMPORTING THE SERVER METADATA IMPORTING EXTENSIONS DATABASES, USERS AND ACCESS RIGHTS IN VIRTUAL DATAPORT DATABASES IN VIRTUAL DATAPORT USER AND ACCESS RIGHTS STRUCTURE IN VIRTUAL DATAPORT User Types Types of Access Rights GRAPHIC ADMINISTRATION OF DATABASES, USERS AND ACCESS RIGHTS Creating Databases Configuration and Deletion of Databases Creating Users Modifying and Deleting Users MONITORING THE VIRTUAL DATAPORT SERVER MONITORING WITH A JAVA MANAGEMENT EXTENSIONS (JMX) AGENT Using JConsole General Information on the Server General Information on Data Sources Information on the Cache Information and Events on Catalog Access (DDL Statements) Information and Events on the Running of DML Statements Information and Events on Transactions DENODO MONITOR Local Monitors Server Monitors Virtual DataPort Queries Monitor Configuring the Denodo Monitor CLUSTER ARCHITECTURES / SERVER BACKUP TOOLS INSTALLATION USE OF THE PING SCRIPT USE OF THE IMPORT/EXPORT SCRIPTS FOR BACKUP AND/OR REPLICATION CONFIGURING SEVERAL INSTANCES OF A VIRTUAL DATAPORT SERVER How to Configure the Slave instances APPENDICES JDBC DRIVERS

6 12.2 MULTIDIMENSIONAL TO RELATIONAL MAPPING USING THE ADVANCED CONDITIONS EDITOR Creating Values for the Conditions Creating Simple Conditions Creating Boolean Conditions Creating Conditions that Use Compound-type Constants CONFIGURING LOGS INSTALLING THE OPENAJAX MASHUP EDITOR INSTALLING THE DENODO SOLUTION FOR MICROSOFT SHAREPOINT TRANSFORMING INCOMING/OUTGOING SOAP MESSAGES WITH XSLT STYLESHEETS JMS CONNECTION DETAILS: JNDI PROPERTIES AND CLIENT JARS Apache ActiveMQ IBM WebSphere MQ OpenJMS Progress SonicMQ INSTALLING THE CONNECTORS FOR SAP ERP, SAP BW AND SAP BI Installing the Connector for SAP ERP (BAPI Data Sources) Installing the Connector for SAP BW and SAP BI (Multidimensional Sources) REFERENCES

7 FIGURES Figure 1 Virtual DataPort Architecture... 3 Figure 2 Authentication screen in DataPort Administration Tool...9 Figure 3 Creating projects Figure 4 Importing a JDBC Data Source Figure 5 Configuring the JDBC connections pool Figure 6 JDBC Data source configuration dialog Figure 7 Schemas, tables and views of a JDBC source Figure 8 Renaming new views with the same name as existing ones Figure 9 Accepting the schema of a base relation Figure 10 Schema of the base view phone_inc Figure 11 Schema of the base view internet_inc Figure 12 Creating a Web service data source Figure 13 Operations and parameters of a Web Service Figure 14 Schema of the base view average_monthly_sales Figure 15 Constructing the union view of phone_inc and Internet_inc Figure 16 Constructing the incidences and average_monthly_sales join view Figure 17 Schema of the incidences_sales join view Figure 18 Constructing the selection view pref_clients_inc_sales Figure 19 Creating a value of type getsumrevenuebytaxids_in Figure 20 Creating aggregation view inc_grouped_by_pref_clients Figure 21 The operation getaveragemonthlyrevenue from the Web Service sales Figure 22 Default base view created for the Web Service operation getaveragemonthlyrevenue.. 43 Figure 23 Flattening the array of the base view average_revenue_array Figure 24 Output schema of the base view average_revenue_array flattened Figure 25 Schema of the flat_revenue view Figure 26 Schema of the view incidences Figure 27 Tree view of incidences_sales Figure 28 Data lineage of incidences_sales Figure 29 Executing the view incidences_sales Figure 30 Result comprised of one tuple with a field of the type array of registers Figure 31 Content of a field of the type array of registers Figure 32 Incidences view execution trace Figure 33 Join view execution trace with errors Figure 34 Confirmation of the replacement of a view Figure 35 View affected by modification to another view Figure 36 Http POST request with XML Body Figure 37 Providing input values to interpolation variables Figure 38 Importing from an ODBC datasource Figure 39 Creating a data source to a multidimensional database Figure 40 Create a base view from an MDX query (multidimensional data source) Figure 41 Importing from an XML data source Figure 42 RSS Xpath Figure 43 Description of an XML source Figure 44 Importing from a JSON datasource Figure 45 Importing a WWW wrapper Figure 46 Importing from a new delimited file source Figure 47 Example 1 of tuple pattern: contents of the delimited file Figure 48 Example 1 of tuple pattern: regular expression to extract the contents of the file Figure 49 Example 2 of tuple pattern: output of the Windows command DIR Figure 50 Example 2 of tuple pattern: regular expression to extract the output of the command DIR Figure 51 Description of a delimited-file-type source Figure 52 Importing a new source of type Google Mini Figure 53 Description of the Google Mini source... 75

8 Figure 54 Base View Creation in Google Enterprise Search / Mini Datasources Figure 55 Creating an Aracne data source Figure 56 Description of the Aracne source Figure 57 Creating base views in Aracne sources Figure 58 Adding an attribute with the most relevant terms of the SEARCHABLECONTENT field Figure 59 Creating an LDAP data source Figure 60 New LDAP data source Figure 61 New LDAP base view from expression Figure 62 Accessible objects of an LDAP base view created from an expression Figure 63 Creating a new BAPI data source Figure 64 Creating a new BAPI (SAP ERP) data source Figure 65 Importing a CUSTOM wrapper Figure 66 New CUSTOM data source Figure 67 Creating a base view from a CUSTOM data source Figure 68 New base view from a CUSTOM data source Figure 69 Creating a base view using an SQL query Figure 70 Editing the value of the interpolation variables Figure 71 Specifying aliases Figure 72 Advanced Configuration screen of the view average_monthly_sales Figure 73 Cache invalidation of a view Figure 74 Editing the Execution Plan of a view Figure 75 Search form for a bookshop Figure 76 Search method for a bookshop Figure 77 Importing a Stored Procedure Figure 78 Properties of a Stored Procedure Figure 79 Execution of the Stored Procedure CalculateAvgRevenue Figure 80 Result of the execution of the stored procedure CalculateAvgRevenue Figure 81 Parameters of a procedure added to a new view Figure 82 Publishing a Web Service Figure 83 Web services Figure 84 Operations created for the view inc_grouped_by_pref_clients Figure 85 Widgets Status Table Figure 86 JSR286 Portlet Figure 87 Portlet search form Figure 88 Adding a new item to the OpenAjax Repository Figure 89 Adding the new widget to the OpenAjax Mashup Editor Figure 90 OpenAjax Widget displaying the content of the incidents_sales view Figure 91 Opening the OpenAjax widget configuration form Figure 92 OpenAjax widget search form Figure 93 Edit Widget Properties dialog: binding properties of a widget with properties of other widgets Figure 94 Binding the property of a widget with the property of another widget Figure 95 Loading the.webpart file into the Web Part gallery Figure 96 Web Part search form Figure 97 Port Configuration Figure 98 Cache configuration Figure 99 Concurrent Requests Configuration Figure 100 HTTP Proxy Configuration Figure 101 Swapping Configuration Figure 102 Configuration of Stored Procedures runtime parameters Figure 103 Response message sent by a JMS listener Figure 104 Response message to a DML query Figure 105 Creating a new JMS listener Figure 106 List of existing JMS listeners Figure 107 Export dialog Figure 108 Creating a new database Figure 109 Description of a database Figure 110 Assigning privileges on a database

9 Figure 111 Creating a normal-type user Figure 112 Creating an LDAP-type user Figure 113 Description of a user Figure 114 Assigning privileges to a user at database level Figure 115 Assigning privileges to a user at view and/or stored procedure level Figure 116 Connection to DataPort from jconsole Figure 117 Virtual DataPort MBeans Figure 118 Attributes tab of the RequestsManagementInfo Mbean Figure 119 Details of a query in jconsole Figure 120 Details of a notification in jconsole Figure 121 Relational star schema Figure 122 MDX query for Visual result mapping (1) Figure 123 Editor of selection conditions Figure 124 Creating a constant value Figure 125 Creating a simple condition Figure 126 RevenueSum schema Figure 127 Compound values editor Figure 128 Creating a value of type getsumrevenuebytaxids_in Figure 129 Condition using compound-type values Figure 130 Result of executing the view RevenueSum Figure 131 SAP Java Connector (JCo) test screen

10 TABLES Table 1 Conversions between DataPort data types and Web service parameter types Table 2 URLs of Web Service s information pages Table 3 Obtaining the XSD Schema of a REST operation Table 4 Invoking a Web Service without parameters Table 5 Invoking a Web Service with one parameter Table 6 Invoking a REST Web Service with an array-type parameter Table 7 Invoking the HTML Web Service with configuration parameters Table 8 Invoking the JSON Web Service with padding (JSONP) Table 9 JDBC Drivers tested with Virtual DataPort Table 10 Contents of a SAP BW base view with Visual result mapping (1) Table 11 Default values for Apache Active MQ Table 12 Default values for IBM WebSphere MQ Table 13 Default JNDI properties for OpenJMS Table 14 Default JNDI properties for Progress SonicMQ

11 PREFACE SCOPE Denodo Virtual DataPort provides business applications with easy access to integrated views of various heterogeneous, distributed and structured and semi-structured data sources. By using examples, this document introduces the reader to the installation, configuration and administration of Denodo Virtual DataPort, including how to create unified views of heterogeneous and distributed data sources. WHO SHOULD USE THIS DOCUMENT This document is aimed at developers and administrators that require a detailed knowledge of how to install, configure and administrate Virtual DataPort, including the creation of unified data views on heterogeneous, distributed, and structured and semi-structured data sources. The detailed information required to develop client applications that access Virtual DataPort is provided in the Developer Guide. SUMMARY OF CONTENTS More specifically, this document: Presents the fundamental concepts of the data integration solutions based on Virtual DataPort. Describes the processes required to install and configure Virtual DataPort. Gives a detailed description, through examples, of how the administration tools of Virtual DataPort are used to construct unified views of data from distributed and heterogeneous sources. Preface i

12 1 INTRODUCTION Nowadays, any organization of a certain importance makes use of a multitude of data sources developed over a period of time. Generally, these data sources are developed using different technologies (relational databases, Web services, XML documents, spreadsheets, flat files, etc.), using very heterogeneous data models, and include both structured and non-structured data. Development of new services for clients and/or optimization of company business processes require the new systems built to integrate the data stored in these inherited data repositories. In addition, it is often needed to access external data sources to interact with partners, providers or customers. The Web itself is also a very valuable information source for many business purposes. Virtual DataPort is a global solution for the real-time integration of heterogeneous, distributed, structured and semi-structured data sources. For this, it combines various features: Integrates and manages the data that are relevant to the company, regardless of its origin and format. It incorporates these data into its data system, in real time or with configurable preloads. Facilitates the construction of distributed information services with high strategic and functional value. This document provides a technical guide to the use of Virtual DataPort. Introduction 1

13 2 GENERAL ARCHITECTURE In outline, Virtual DataPort enables business applications to process a series of distributed and heterogeneous data sources, including external sources, as though the data were contained in a large Virtual Database. This virtual database allows creating views that combine and integrate said data. Virtual DataPort acts as a mediator that provides a structured and unified view of the data contained in all the data sources included in the system. The system can easily deal with a wide range of structured, semi-structured and unstructured data sources such as: databases, Web sites, spreadsheets, XML documents, Web Services, LDAP servers, flat text files, indices on unstructured information, etc. The system allows easily importing each data source into the virtual database. Then, a SQL-like language called Denodo VQL (Virtual Query Language) is used to create views that arbitrarily combine the data of said relations using operations such as selections, projections, unions, joins, groups, etc, thus creating unified views of the source data. In this process, Virtual DataPort is able to work with sources that have limited query capacity (for example, in many Web services only queries specifying certain mandatory parameters are allowed). Furthermore, the Virtual DataPort query engine is capable of querying non-structured data and combining it with structured and semi-structured data. When the system receives a VQL query on a previously defined view, it can generate an execution plan for the query, which consists of a list of subqueries that are sent in real-time to the various sources involved and a series of operations combining the data obtained from each source. Furthermore, Virtual DataPort incorporates a system (called cache module) which allows the administrator to decide on the mechanism to be used for accessing the source data: The system can access the source data in real time, thus providing totally updated data. Caches can be created and configured for the sources or views as required. Virtual DataPort also allows the updating of data sources, provided that these are capable of supporting transactions. The data integration system provided by Virtual DataPort is modeled on three separate layers: the user layer, the logical layer and the physical layer (wrappers). Figure 1 provides a general overview of the system architecture. Each of these layers is described below. General Architecture 2

14 Figure 1 Virtual DataPort Architecture 2.1 PHYSICAL LAYER The physical layer abstracts the higher architecture layers of the difficulties of interacting with the data sources. It also provides a vision of the data sources according to a common metamodel. These tasks are carried out through the so-called wrappers. A wrapper extracts data from a source, interprets the results obtained and returns them to the system in the format required by Virtual DataPort metamodel. Furthermore, where permitted by the source, a wrapper can also insert, update and/or delete information in a source. The wrappers allow Virtual DataPort to process all the external sources in a consistent manner, without concern for their specific characteristics. Virtual DataPort directly supports the following types of wrappers: Relational databases: They extract data from a Remote Database via JDBC or ODBC. Furthermore, where permitted by the source, a wrapper inserts, updates and/or deletes information in a source. Multidimensional databases: They extract data from multidimensional databases such as SAP BW and SAP BI. General Architecture 3

15 Web Services: They extract data invoking operations provided by SOAP Web services. XML: They allow extracting data encapsulated in XML files. Those files can optionally be validated using a specific DTD or schema. XML documents can be accessed on the local drive or remotely accessed via protocols such as http or FTP. A common use of this type of wrapper is the importing of REST-type Web services (including RSS and ATOM web services). JSON: Extract data from documents in JSON format [JSON]. They are useful to access REST-type Web services returning their output in JSON format. Flat files: Extract data from flat files in CSV format (Comma Separated Values) or similar. CSV documents can be accessed on the local drive or on remote locations via http or FTP. Web: Provide access to semi-structured data contained in websites. For instance, these wrappers can automate the process of querying a web form and extracting the results. These wrappers are generated with Denodo ITPilot [ITPILOT]. Aracne: Denodo Aracne [ARCN] allows for the crawling, indexing and search for non-structured data contained in repositories such as Websites, relational databases, local file systems and servers. Indexes created using Denodo Aracne may be imported directly to Virtual DataPort for querying and combining with structured and semi-structured data. Google Enterprise Search / Google Mini: Google Enterprise / Google Mini [GMINI] is Google s corporate solution for crawling, indexing and searching websites. The indexes created using Google Mini may be imported directly into Denodo Virtual DataPort for querying and combining them with structured and semi-structured data. LDAP: Extracts data contained in LDAP directories such as Microsoft Windows Server Active Directory [MS-AD]. BAPI: They invoke SAP BAPIs (Business Application Programming Interfaces) to extract data stored in SAP ERP and other SAP applications. CUSTOM (also named as MY wrappers): They extract data from a source through a Java implementation provided by the Virtual DataPort administrator. Virtual DataPort provides an API, so users can create their own wrappers for specific sources. The CUSTOM wrappers also allow inserting, updating and/or deleting data from the sources. Wrappers for databases, Web services, flat files, Aracne, Google Mini and XML are automatically created by Virtual DataPort as the corresponding data sources are imported (see 5.2). The wrappers for semi-structured Web sources can be created with Denodo ITPilot [ITPILOT]. 2.2 LOGICAL LAYER The logical layer integrates and combines the relations exported by the different wrappers (called base relations or base views) to create the views that will comprise the system global schema. General Architecture 4

16 Once the base relations representing the system sources have been created, the administrator can create views that intercombine them as required, thus creating the global schema views (or derived views). It is important to point out that this process can be carried out in a recursive manner in several steps: a derived view can be used as a base to create new views, thus allowing combinations of arbitrary complexity. Views are defined using the Denodo VQL language [VQL] although, as explained in the next sections, the administration tool allows graphically creating them, so the VQL statements do not have to be manually written. Once the views of the global schema have been created by combining source data, the logical layer is capable of responding to queries expressed in VQL both on derived views and on base relations. The VQL query language is SQL-based, but it incorporates different extensions to handle heterogeneous and distributed data. For example, VQL includes certain commands to allow querying non-structured data and combining it with structured data. It also supports compound types such as arrays and registers. When the system receives a query, it checks that it can be resolved depending on the query capabilities supported by the data sources, it then draws up the possible execution plans, selects the most suitable one and executes the query returning the results obtained to the higher layer. The logical layer of Virtual DataPort also allows writing to data sources using INSERT/UPDATE/DELETE operations, provided that these are capable of supporting transactions. The following modules can be differentiated in the logical layer: Query Plan Generator: Firstly, the plan generator decides if the query received can or cannot be answered in accordance with the query capabilities supported by the data sources. Where it is possible, it generates the possible execution plans for the query. Optimizer: Aims to select the optimum execution plan from all the options (obtained by the Query Plan Generator). The query capabilities of the sources are also considered so, when possible, the processing of some operations is delegated to the data sources, thus achieving more efficient execution and less data exchange through the network. Other aspects taken into account are the most optimal execution strategies for join operations. Query Execution Engine: Once the optimum plan has been selected, the execution engine is responsible for putting it into practice, executing the necessary subqueries on the sources and integrating the results obtained to generate the global response. In turn, the execution engine takes into consideration that information from the sources which is already preloaded in the cache module, whereby unnecessary access to data sources is avoided, thus achieving greater efficiency Data Module: Cache As mentioned above, Virtual DataPort has a system (called cache module) to store local copies of the source data as required. This cache is stored in a relational database accessible through JDBC. Virtual DataPort embeds an Apache Derby [DERBY] database that can be used to store the cache information. It is also possible to use the following external DBMSs to store the cache: MySQL [MYSQL], Microsoft SQL Server [MS_SQL]and Oracle [ORCL]. The system will generate and maintain a table in the cache database for each base view or view for which the cache has been enabled. As the queries are made, the tables are filled with the rows obtained from the sources. Each view in the cache has an associated expiration time, whereby the expired tuples are not considered in the queries and are automatically deleted at regular intervals. The cache can be preloaded periodically by writing a query that retrieves all the required data and scheduling this query to be repeated at the desired time interval. General Architecture 5

17 The cache can also be disabled for a specific query or for the desired views, whereby it is always possible to access the data in the sources in real time. The section explains how to enable the cache module. 2.3 USER LAYER The user layer implements the interface between the client applications and the Virtual DataPort server. Thus, its responsibility is to provide client applications with means to establish a connection with the server, send queries and obtain the results. When the logical layer returns a response to the query, this layer is also responsible for providing it to the client application through the connection interfaces defined to this effect (see Virtual DataPort Developer Guide [DEV]). Denodo Virtual DataPort can be accessed via JDBC, ODBC, SOAP, REST, JSON and RSS Web Services (see section 6), JSR-168/286 Portlets, Open Ajax widgets and Microsoft Web Parts (see section 7), and through Virtual DataPort JAVA native API. General Architecture 6

18 3 INSTALLATION AND INITIAL CONFIGURATION The Denodo Platform Installation Guide document [DENINST] provides all the needed information to install Virtual DataPort, including the minimum hardware and software pre-requisites. It also includes instructions for using the installation tool and to perform the initial system configuration tasks. Installation and Initial Configuration 7

19 4 EXECUTION To start and stop the Virtual DataPort server, there are two options: The Denodo Platform Control Center (see Denodo Platform Installation Guide [DENINST]) allows, among other functionality, to start and stop all servers and tools of the Denodo Platform. Executing the script: $DENODO_HOME/bin/vqlserver.sh startup (vqlserver.bat startup in Windows) to start the server. To stop it, execute $DENODO_HOME/bin/vqlserver.sh shutdown (vqlserver.bat shutdown in Windows). To start the Virtual DataPort administration tool, there are two options; 1. Using the shortcut included in the Denodo Platform Control Center [DENINST]. 2. Using the script: $DENODO_HOME/bin/vdpadmin.sh (vdpadmin.bat or vdpadmin.exe in Windows). 4.1 EXECUTING THE VIRTUAL DATAPORT GRAPHIC ADMINISTRATION TOOL Once the Virtual DataPort graphic administration tool has been launched, the first step is authenticating in the DataPort server (see Figure 2). The server uri format is: //name_machine:port/database (e.g. //localhost:9999/admin) where: name_machine: name of the machine in which the server resides. port: number of the server execution port. database: (optional) Virtual DataPort database to which you wish to connect. When a Virtual DataPort server is installed, a default database called admin is created. Execution 8

20 Figure 2 Authentication screen in DataPort Administration Tool For more details on the concepts of databases and users in Virtual DataPort, see section 9. If this is the first access to the server, the default administrator user can be used (user login admin with password admin ). Users should change their password after connecting to the server for the first time. For this, open the Change password dialog on the File menu. There are two modes for connecting to the server: database administration mode and server administration mode. The database administration mode is accessed when the server uri specifies a database and, in addition, the specified user is not an administrator. This mode allows managing the data sources, base relations and unified views that form part of a specific database of the DataPort server. The server administration mode is reserved for administrator users and, in that case, it is not mandatory that the supplied uri specifies a database. This mode allows executing tasks like those listed below: Create, modify or delete databases (see section 9). Create, modify or delete users and manage their access privileges (see section 9). Configure the cache system (see section 8.1.3). Configure the swapping policy (see section 5.6.3). Configure the server thread pool (see section 8.1.2). Configure the server execution and stop ports (see section 8.1.1). Configure the internationalization options (see section 8.1.8). Configure other preferences such as HTTP proxy preferences (section 8.1.5), runtime parameters for stored procedures (section 8.1.7) and limiting the number of concurrent requests processed by the server (section 8.1.4), Configure JMS listeners to receive queries from a JMS server (see section 8.1.9) If the specified URI includes a database and the user is an administrator, then the administration tool will allow her to manage both the server and the specified database. Execution 9

21 4.2 DOCUMENTATION AND ONLINE HELP Online help for the Virtual DataPort administration tool can be accessed by choosing the Online Help option on the Help menu. It is also possible to access contextual help information about the current dialog by clicking on the? icon on the upper-right section of the screen. The online help is organized in topics according to the table of contents shown on the left side of the Help screen. It is also possible to perform text-based searches on the help contents by using the Find tab. The product manuals are also available. They can be accessed selecting the option Denodo Platform Documentation in the Help menu (a new window browser will be opened) 4.3 MAIN AREAS OF THE ADMINISTRATION TOOL The Virtual DataPort Administration tool has three main areas: the menu toolbar, the task pane and the workspace. The menu toolbar is placed on the top of the screen and includes the following options: The File menu allows closing the session ( Disconnect option), exiting the tool, changing the password of the active user, importing or exporting metadata for creating/restoring backups (see section 8.2), importing new extensions to the tool (see section 8.3), and refreshing the metadata from the catalog. Refreshing is useful when creating new elements in the catalog by directly using the VQL language, instead of creating them graphically. The View menu allows configuring the administration tool preferences (see section 4.4) and managing the list of hidden views. This list contains views that are not shown in the menus of the task pane on the left of the screen because they have been manually hidden by the user (see section 5.2.1). Clicking on one of the hidden views will make it visible again. To hide a view, simply right-click on its name on the task pane and choose the Hide view option. The Tools menu allows selectively invalidating the server cache (see section 4.5) and searching the DataPort metadata catalog (see section 4.6.2). The Help menu allows accessing the online help and the online manuals (see section 4.2). The task pane is composed of a list of sections, each one containing a hierarchy of drop-down menus. The available sections are: Project Management. It allows creating and removing projects (see section 5.2). Connect. It allows importing data sources and creating base views (see sections 5.3 and 5.5). Combine. It allows creating new views by combining other views through operations such as union, join, projection/selection, etc. It also allows editing existing views, and examining the dependencies between them. See section 5.4. Publish. It allows publishing views and/or stored procedures as Web services (SOAP, REST, JSON, RSS, ) and / or visual widgets (JSR-168/286, Microsoft Web Part, ). See section 7. VQL Shell. It allows manually writing and executing VQL statements. See section For administrator users, the task pane also has the following sections: Database Management. It allows managing the databases of the DataPort server. See section 9. Execution 10

22 User Management. It allows managing users and access rights. See section 9. Server configuration. It allows configuring different aspects of the server. See section 8. The menu tool bar also contains the Quick Search drop down list (see section 4.6.1). This list is a shortcut for the menus in the task pane. For example, if the current section is Combine, the task pane shows the list of Base Views, Views and Stored Procedures in each project. Therefore, the Quick Search drop-down list will contain the same elements and selecting one of them will have exactly the same behavior than selecting it on the task pane. 4.4 TOOL PREFERENCES The View-Preferences menu of the graphical administration tool allows configuring certain preferences described in the following subsections Locale The Locale configuration of the Administration Tool controls the appearance of data that depends on the user s region. I.e. the formatting used to display the fields of type date. This can be changed in the dialog of the View Preferences Locale menu. Note: This preference does not change the internationalization preferences of the Virtual DataPort server. See section for information about changing this preference of the server Connection Parameters The connection between the Administration Tool and the Virtual DataPort server can be configured in the View Preferences Connection dialog. It is possible to configure the following parameters: Query Timeout: Maximum time (in milliseconds) the Administration Tool will wait for a query to finish. If 0, it will wait indefinitely until the sentence ends. Chunk Size: The results obtained by executing a sentence can be divided into blocks (chunks), so the Server does not have to wait until a sentence ends, to send the processed tuples to the Tool. This parameter establishes the maximum number of results that a block can contain. When the Virtual DataPort server obtains enough results to complete a block, it sends this block to the client and continues processing the next results. Chunk Timeout: This parameter establishes the maximum time (in milliseconds) the Server waits before returning a new block to the Tool. When this time is surpassed, the Server sends the current block even if it does not contain the number of results specified in the Chunk Size parameter. Note: if the values of Chunk Size and Chunk Timeout are 0, the Server returns all the results in a single block. If both values are different than 0, the Server returns a chunk whenever one of these conditions happen first: o The chunk is filled (Chunk Size) o Or, after a certain time of not sending any chunk to the client (ChunkTime). Default Server URL: The default Server URL that appears in the authentication dialog of the Administration Tool. Execution 11

23 4.5 INVALIDATE CACHE This option lets the user delete the content that is currently stored in the cache for one or more views. Open the Invalidate cache dialog by clicking on the menu Tools Invalidate Cache. Invalidate the cache content of one or more views by selecting them and clicking the Invalidate button. Check the Invalidate on cascade option to invalidate all the views underlying this current view. Example: if there is a view named V3, which is a projection of the base views BV1, and BV2 and you Invalidate the cache on cascade of the view V3, automatically the cached content of the views BV1 and BV2 will be also automatically invalidated. 4.6 SEARCH A user can search an element of the Virtual DataPort catalog using: Quick Search: to easily search by name. Catalog Search: to search objects in the catalog using more advanced filters such as Element Type, Creation Date, Description, etc Quick Search The easiest way to open an element is searching its name in the Quick Search drop-down list located at the top of the Administration Tool and then clicking on its name. The elements accessible from the Quick Search drop-down list are exactly the same as the ones shown in the pane on the left side of the tool. Depending on the current section, the elements shown may vary. For example, if the current section is Combine, the panel on the left contains a list of Base Views, Views and Stored Procedures. Therefore, the Quick Search drop-down list will contain the same elements Catalog Search The Catalog Search tool searches all the elements in the catalog of the current Virtual DataPort server database. It is more powerful than Quick Search because it allows the user to search, not only by name, but also by type, owner, description, etc. Catalog Search can be accessed by selecting the option Catalog Search in the Tools menu. The possible criteria for searching elements are the following: Element Type. If the selected element type is Views, the user can also select View Type, Swap active and Cache Status as search criteria. Owner. User name of the user that created the elements. Last User Modifier. User name of the last user that modified the elements. Initial Creation Date and End Creation Date. Elements that were created between the Initial Creation Date and the End Creation Date. Initial Modification Date and End Modification Date. Elements that were modified between the Initial Modification Date and the End Modification Date. Execution 12

24 Description. Elements that contain the value of this field in its description. It is possible to open the configuration of an element included in the results by clicking on its name. 4.7 QUERY MONITOR The Query Monitor lists the queries that the Virtual DataPort server is currently executing. If we select a row of the table, the Tool displays the Tree View of the query. That is, the tree of views used to form the results of the query. Click on each node of the tree to display information about it, such as number of processed rows, state, etc. You can copy the value of each parameter by right-clicking on it and then, clicking on Copy. To store the current tree of views into an image file, click on Save image. The Monitor updates automatically the list of queries unless we deselect the option Automatically refresh every In this case, we can click on Update to update the list. To stop the execution of a query, select its row in the table and click on Stop query. Execution 13

25 5 VISUAL CREATION OF VIEWS This section describes how to use the Virtual DataPort graphic administration tool to create unified data views graphically from various heterogeneous sources. We will use the following example as a guide when describing the process: Example: Unified data about customer sales and incidents. A telecommunications company offers phone and internet services to its clients. Data on the incidents reported in the phone service are stored in a relational database, which is accessed through JDBC. And data on the incidents reported in the Internet service are stored in another relational database also accessed through JDBC. In our example, the director of the I.T department wants to monitor the number of incidents (either telephony or Internet) notified by the clients with the greatest sales volume to establish whether or not measures should be taken to increase client satisfaction. Data on customer sales volumes are managed by another department of the company. That department provides a Web Service so the other departments can access to that data. In this example, we will see how Virtual DataPort can be used to construct a unified data view to meet the needs of the I.T department, by obtaining the total number of incidents from clients with the greatest sales volumes. The path DENODO_HOME/samples/vdp/incidences contains: SQL scripts for creating the tables used in the examples of this manual (version for MySQL, Oracle and PostgreSQL). Note: to follow the examples of this guide, use one of these scripts to create the required tables in a database. A.war file with the implementation of the Web Service used in the examples. It has been tested with Tomcat 5.x [TOM] and Axis 1.x [AXIS]. A WSDL file with the description of the Web service used. vql scripts (VQL stands for Virtual Query Language [VQL]) that create the objects (data sources, views, stored procedures ) that we will learn to create in this manual. You don t need to use them if you are going to follow this guide. Otherwise, edit the vql script that matches your database: with the host name of your database. In the CREATE WRAPPER JDBC commands, change the parameter RELATIONNAME, so it matches the name of the schema in your database. I.e: change RELATIONNAME='VDB.INTERNET_INC' for RELATIONNAME= test.internet_inc'. 5.1 ACCESS TO THE GRAPHIC ADMINISTRATION TOOL The first step is to access the tool in database administration mode (see section 4.1). If a database and users have not been created yet (see section 9), you can use the username (admin) to connect to the default database (vdp). Visual Creation of Views 14

26 5.2 CREATING AND MANAGING PROJECTS The different components (data sources, base relations, views, stored procedures, web services and widgets) that can be created using the Virtual DataPort administration tool are grouped together into projects. On starting the tool for the first time, there will be only one project available called Default. To create a new project, select the Project Management tab and click on the option new (see Figure 3). On this screen, give a name to the project and, optionally, a description. The new project will be created on clicking ok. In our example, the name of the new project is VDPExample. Figure 3 Creating projects A project can be edited by clicking on its name in the drop-down menu on the left of the screen or deleted by clicking on the drop option. If a project is deleted, the elements it contains are automatically moved to the Default project. NOTE: these projects only exist in the graphical administration tool used to create them Moving Elements to Other Projects As we will see in the next sections, on creating a new datasource or a new view, the new element can be allocated to one of the existing projects. To move an existing element to another project, proceed as follows: 1. Expand the menus on the left of the screen until the required element has been located. The data sources will be accessible from the Connect tab and the views, from the Combine tab. 2. Right-click on the element to be moved and use the Move to Project option to select the destination project. Visual Creation of Views 15

27 5.3 IMPORTING DATA SOURCES AND CREATING BASE VIEWS The first step in our example is to import the data sources and create base views (also called base relations) which represent the data necessary to construct the unified views in Virtual DataPort. In the following sections, we will deal with: Importing Database sources and creating base views from their tables, which will allow us to import the databases of incidents of our example (sections and 5.3.3). Importing Web-Service-type sources and creating base views from their operations (sections and 5.3.6), that will be used to import data on sales. Deleting data sources (section 5.3.2). Examining the schema of views (section 5.3.4). How to import other types of data sources and how to create base views from them is explained in section Importing JDBC-Type Sources To create a new DataSource, open the Connect tab on the left side of the tool. Then, click on the project to which the new source is to be added. In our example, select the VDPExample project. When the Relational DB menu is dropped down, it displays the list of external databases added as sources to the Virtual DataPort database. They appear classified in two categories: JCBC and ODBC. To add a new JDBC database to the list, click on the New button beside the JDBC database list. You then access the screen for creating a new JDBC data source (see Figure 4). The following data are then requested in this dialog: Name. Name of the new data source in Virtual DataPort. In our example, we will use the name phone_ds for the data source of the telephone service table. And internet_ds for the data source of the Internet service incidents table. Database Adapter. In this selectable it is possible to choose one of the DBMSs that Virtual DataPort includes a specific adapter for. A Generic adapter is also included for using with JDBC-accessible DBMSs without a specific adapter. Each specific adapter implements several optimizations for its target DBMS, so they should be used whenever possible. If the Choose automatically option is selected (recommended), DataPort will discover the most suitable adapter to connect to the DBMS. On selecting an adapter, the Driver Class field is completed automatically and the DB Uri field is completed with a template of the uri needed to connect to the selected database. The Appendix 12.1 contains a list of the JDBC drivers included and/or normally used. Driver JAR file. Path to the.jar file containing the JDBC driver to be used (optional). When using a JDBC driver not included with Virtual DataPort, click Browse to select the.jar file containing the required driver. Note: This is only necessary if the driver is not included with Virtual DataPort and the driver has not been included in the CLASSPATH of the server. To include a new driver in the classpath, place its.jar file in the DENODO_HOME/extensions/thirdparty/lib path (DENODO_HOME is the Denodo Platform installation directory) and restart the server. Driver class. Name of the JAVA class of the JDBC driver to be used. For example, for a MySQL database the value com.mysql.jdbc.driver is normally used. The appendix 12.1 contains a list of the values that must be used with the JDBC drivers that are included and/or most commonly used with Virtual Visual Creation of Views 16

28 DataPort. DB URI. Access uri to the database. In our example, we assume that both tables of incidents are stored in the database incidents_center of a MySQL server located in the machine acme. So, the URI is: jdbc:mysql://acme/incidents_center. The appendix 12.1 contains a list of the values that must be used with the JDBC drivers that are included and/or most commonly used with DataPort. Login. User login to access the external database. Password. User password to access the external database. If the Choose automatically option is checked (recommended), the system will connect to the DBMS to discover the most suitable adapter. On marking this option the Test Connection option is automatically selected. Finally, if the Test Connection option is selected, the system will connect to the source to verify it can be accessed from the DataPort server. Figure 4 Importing a JDBC Data Source Optionally, we can add properties that will be used as arguments when establishing a connection with the database. To do this, click on Driver properties Note: if this data source will connect to Oracle and we want the data source to list its synonyms, we have to add the property includesynonyms = true. Open the Connections Pool Configuration dialog to configure several parameters of the connections pool that Virtual DataPort will use to optimize access to the external database (see Figure 5): Visual Creation of Views 17

29 Initial Size: number of connections with which the pool is to be initialized. A number of connections are established and created in idle state, ready to be used. Max Active: maximum number of active connections the pool can manage at the same time (a negative value implies there is no limit). Ping Query: SQL query used by the pool to verify the status of the connections. The query should consume as little resources in the database as possible, and the table in question should exist. Test Connection: if this property is checked and there is a specified ping query, each connection retrieved from the connection pool will be validated by executing the ping query. Note: in production environments, we strongly recommend selecting the Test connection checkbox and defining a Ping query. Figure 5 Configuring the JDBC connections pool After creating the data source, we can go back to its configuration dialog by clicking on the data source in the leftside menu and then on the Edit button. In the Edit JDBC Data Source, we can configure the specific behavior of the JDBC data source such as: Which operations are delegated to the database (executed in the database instead of in Virtual DataPort Server) Which operations this data source has support for. See section for more details about this dialog. Note: In most cases, the default value for these options is correct. Therefore, this dialog will only be useful in very particular environments. Visual Creation of Views 18

30 Figure 6 JDBC Data source configuration dialog Deleting a Data Source Once a data source has been added, it can be deleted it by clicking on the drop button beside the data source name in the Connect tab. If there are other elements in the DataPort catalog (e.g. views) which depend on that data source, the Tool will display a warning message informing the user that the operation will be executed on cascade. If the user decides to proceed, the elements having dependencies with that data source will also be deleted. This operation will fail if the user does not have the required privileges to delete these elements Creating Base Views from a JDBC Data Source After creating a data source, it can be used to create base views. For this, you should click on the data source in the list on the left side of the screen to access its contents. In the case of JDBC or ODBC data sources, a tree will be shown with the schemas of the external database. Click on any schema to inspect its tables and their fields (see Figure 7). Visual Creation of Views 19

31 Follow these steps to create one or more base views: Figure 7 Schemas, tables and views of a JDBC source 1. Select the check boxes beside the tables you want to create base views from. 2. Select the project of the new base views using the Target Project drop-down. 3. Click on the Create base views button beside the Target Project drop-down. After this, the new base views are listed on the left side of the screen hanging from its JDBC data source. In our example, we will create a base view for the phone incidents table phone_inc and another for the internet incidents table internet_inc. We will select VDPExample as the target project. phone_inc provides access to data on the incidents registered in the phone service. Its fields are: the Tax ID (taxid) of the client that reported the incident, an incident identifier (pinc_id), a description (description), the time at which it occurred (ttime), an incident type (inc_type) and an additional specific field (specific_field3). The attributes of internet_inc are the Tax ID (taxid) of the client that reported the incident, an incident identifier (iinc_id), a description (summary), the moment at which it occurred (ttime) and another two additional specific fields (specific_field1 and specific_field2). When selecting two or more tables to create base views, you may need to provide a new name for the views that have the same name as other existing views (see Figure 8) Visual Creation of Views 20

32 Figure 8 Renaming new views with the same name as existing ones When importing only one table at a time (see Figure 9), you have more control over the creation of the base view because you can: Change the name of the new base view. Change the name and type of the new base view s attributes. Select the project of the new base view using the Target Project drop-down. Visual Creation of Views 21

33 Figure 9 Accepting the schema of a base relation Note: It is possible to create a base view from a SQL query instead of from a table ( Create Base View from Query button). This option is explained in section Viewing the Schema of a Base View By clicking on a view we have created, we can access the dialog that shows its schema in Virtual DataPort (see Figure 10 and Figure 11 with the schemas of the base views generated from phone_inc and internet_inc). Click on Edit to open the Edit view dialog. For base views, this dialog allows: Renaming the view. Renaming the view s attributes. Changing the types of the view s attributes. For derived views, the dialog corresponds to the operation used to create the view (i.e. join, groupby, union, etc). In it, you can: Rename the view. Modify its specific parameters. For example, for join conditions you can change the join conditions, projected fields, etc. Alternatively, you can replace the view with another one with the same name (see section for more details about edition/replacement of views). Other possible actions from this screen are: Execute: Displays the Execution view dialog where the user can execute the default query select * from viewname or add conditions to the WHERE clause. See section for more details about execution of queries. Visual Creation of Views 22

34 Delete: Delete the view. If there are other derived views using the view in their definition, the tool will warn the administrator that deleting the view will cause the deletion on cascade of the derived views. View VQL: See the VQL code generated to create the view. See section Advanced: Opens a dialog to change advanced settings of the view. See section 5.6. Used by : Displays a diagram with the views that depend on the current one. See section Figure 10 Schema of the base view phone_inc Visual Creation of Views 23

35 Figure 11 Schema of the base view internet_inc View VQL This dialog displays the VQL code generated to create the view. The following options control how the VQL code of the view is generated: Show dependencies. If selected, the Tool displays the VQL code to create the view and the elements that the view depends on. Otherwise, it shows only the code to create the view but not the elements that it depends on. For example, if we are viewing the VQL of a view and this option is selected, the Tool will display the VQL code to create the data sources that this view depends on. Drop elements. If selected, all the CREATE statements are preceded by a DROP CASCADE statement. Therefore, the generated VQL code will delete, if exists, the element with that name and all the elements that depend on it. E.g. when a server imports a JDBC data source ds_jdbc_acme, first it will delete the existing JDBC data source ds_jdbc_acme and all the views that depend on this data source. Replace existing elements. If selected, the statements to create elements will be like this: CREATE OR REPLACE As a result, the generated VQL code does not delete the elements that depend on the one that the CREATE statement is creating. E.g. when importing an LDAP data source ds_ldap_activedirectory and the server already has an LDAP data source with that name, the Server will replace this data source with the new one. However, it will not modify or delete the views that depend on the existing data source. Visual Creation of Views 24

36 Do not replace existing elements. If selected, the statements to create elements will be like this: CREATE As a result, if there is already an element with the same name and type, the existing element will not be replaced nor deleted and the statement will fail. If you want to copy the VQL code, you can click on Copy to clipboard to copy all the code of the view. Or, just select the code that you are interested in, right-click it and click on Copy Used By This dialog displays a diagram with all the views that depend on the current one. On the left side of the dialog, there is a list of all the top-level views that use the current one in their definition. Toplevel views are those that are not used to define any other view. Click on one view to display how it is constructed: Click on the tree nodes representing combining operations (joins, unions, selections, etc) to display their main properties. Click on a data source to view its details. Click on a view or a base view to open the dialog showing its schema. Click on Save image to export this tree into an image file Importing Web-Service-Type Sources Continuing with our example, the next step is to create the data source that obtains the sales data. As mentioned earlier, this source is of the Web Service type. To create a new data source representing a Web Service, open the Connect Projects VDPExample tab and click on New beside the Web Services tab. The new Web Service data source dialog will be displayed (see Figure 12). The following data are requested: Name: Name of the new data source. In our example, the name of the new data source is sales. WSDL: Path to the WSDL file containing the Web service specification. We can provide a URL or select a local file by clicking on Browse. In our example, the path is $DENODO_HOME/samples/vdp/incidences/sales.wsdl. End Point: URL to access the Web service. E.g. Usually, the WSDL contains this URL, but sometimes it does not or it does not point to the right location. o o From WSDL: The end point is obtained from the WSDL. From Variable: The end point is obtained from an interpolation variable. This means that the base views created over this data source will have an extra field with the name specified in the field Variable. When querying these views, we have to indicate the end point of the Web service in the WHERE clause of the query. For example, SELECT * FROM v_revenue WHERE endpoint = ' This option is useful when the end point changes regularly or is obtained from another source at Visual Creation of Views 25

37 runtime. See section for more information about interpolation variables. o Specify: Indicate the URL of the end point in the text field below. This option is useful when the WSDL does not contain and end point, or it is not correct. E.g. if we deploy the sales Web service in a Web container that does not listen to connections at localhost:8080. Authentication: Virtual DataPort can access Web services that require authentication. It supports five authentication protocols: o HTTP Basic and HTTP Digest. [HTTP-AUTH] describes these authentication methods. o o HTTP NTLM. Uses the Microsoft's NT LAN Manager (NTLM) Authentication Protocol [MS- NLMP] to access Microsoft Windows servers. Virtual DataPort supports NTLM v1 and NTLM v2. WSS Basic and WSS Digest. Web Services Security [WSS] is a standard for the implementation of security features in applications using Web services. Currently, Denodo supports the authentication profile Username Token [WSS-UT]. Instead of providing the credentials when creating the data source, we can assign interpolation variables (see section ) to the credentials and supply them at runtime. To do that, select the box Obtain credentials from variables and fill in the text fields below ( Name variable and Test variable ) E.g., we select HTTP Basic authentication, check the box Obtain credentials from variables and fill in the field Login Name variable with user_name and Password Name variable with password. The views created over this data source will have two extra fields called user_name and password. When querying these views, we have to provide their value in the WHERE clause of the query: SELECT * FROM average_monthly_array WHERE user_name='admin' and password='admin'. The values of the fields Test value are used to retrieve the WSDL from the server. If the HTTP connection with the server is secure (HTTPS), select the Check certificates box to accept only the certificates that the Java Virtual Machine considers valid. Otherwise, every certificate will be accepted. See the keytool reference [KEYTOOL] for details on how to import an SSL certificate into the Sun Microsystems JVM. Proxy: If the access to the Web Services is established through a proxy, the host name and port where the proxy is executed must be provided. If it is an authenticated proxy, the user identifier and a valid password must be also specified. It is also possible to use the default http proxy configuration of the server (see section 8.1.5) by selecting the option Default. Visual Creation of Views 26

38 Figure 12 Creating a Web service data source Creating Base Views from a Web Service Once a Web Service data source has been added to the system, it can be used to create base views. For this, you should click on the data source on the list on the left side of the screen to access its contents. In the case of Web-Service-type sources, the tool will show the list of Web Services contained in the WSDL file and the various ports that make up each of them. For each port, the tool will show the list of operations included in it. For each operation, its input parameters, output parameters and input/output parameters are shown (see Figure 13). In our example, the sales Web service has one port with several operations. The operation we are interested in is getaveragemonthlyrevenuebytaxid. It has an input parameter called in0 that specifies the client Tax ID (taxid) and returns the average monthly volume of sales. There is a Create Base View button beside each Web Service operation. If this button is clicked, the system generates a base view from the selected operation. The name of the base view and the name and type of its attributes may be modified at this stage. The project to which the new view will be allocated can be specified using the drop-down Target Project. In our example, select the VDPExample project. In our example, we will rename the base view to average_monthly_sales, the field in0 to taxid and the field return to revenue. Visual Creation of Views 27

39 Figure 13 Operations and parameters of a Web Service Click Ok to create the base view. If the user wishes to create a base view and then import another one from the same data source, he/she can press the OK and import new view button, which will create the view and will go back to the screen where the Web Service operation lists is shown. After creating the new base view, it is listed on the left side of the screen below the Web-Service-type data source used to create it. Clicking on the base view, we can access the page that shows its schema in Virtual DataPort (see Figure 14). The available options in this screen were described in section The results returned by an operation may be of compound data type (e.g. arrays of elements). In that case, these results will be represented in the base view with array and register compound-type fields. It is possible to flatten these fields to achieve a more appropriate granularity in the tuples of the base view. This process is described in section Another similar situation occurs when an operation imported from a Web Service has a compound-type input parameter (e.g. an array of elements). In this case, the input parameters will also be represented in the base relation with array and register compound types. Furthermore, the ROW constructors (to create register-type elements) must be used to build query conditions on this base relation and { } to create type elements. For example: { ROW( parameter1, parameter2 ) } specifies one array containing one record with two fields, where parameter1 is the value of the first field and parameter2 is the value of the second field. Visual Creation of Views 28

40 Figure 14 Schema of the base view average_monthly_sales 5.4 VISUAL CREATION OF VIEWS AND QUERIES Once the base views have been created, you can access the Combine tab to create derived views that combine and integrate data from different base views. After they have been created, derived views can then be used to construct new views or queries. The following sections describe the process of creating views using our example to illustrate the process Creating Union-Type Views A union-type view allows merging the tuples from various input relations in a single view. In standard relational algebra, all the relations must have the same schema (same attributes with the same type). However, in Virtual DataPort an extended union operation is used, whereby if any of the input relations has an attribute that is not present in the others, this is added to the resulting view. To define a union view, open the Toolbox menu and click on Union. To select the relations that will be part of the union, drag&drop from the list of views and base relations that appear on the left side of the screen or click the name of the view in the Quick Search panel on the top of the screen. In our example, we will drag&drop the base views phone_inc and internet_inc (see Figure 15). As input views are added to the union view, the tool automatically generates the schema of the resulting union view. The schemas of the input views are shown on the left of the work area and the output schema is displayed on the right. Visual Creation of Views 29

41 Figure 15 Constructing the union view of phone_inc and Internet_inc Below the input views, the Associations list is shown. The fields united by an association are considered a sole attribute in the output schema. In order to establish an association between two fields they must be of the same type. By default, the system will create associations between the attributes of the same name and type (in our example this occurs with the fields time and taxid). Additional associations may also be specified between attributes which receive different names in the different input views. In our example, we will do this with the attributes description of phone_inc and summary of internet_inc. For this, drag&drop from one attribute to another and one sole attribute will appear in the output schema representing both. The default name assigned to the attribute in the output schema will be that of the first attribute marked in the drag&drop operation. Any association can be deleted using the [x] button beside the association. The following actions can be carried out on the output view schema (right side of the screen): Change the name of the new view Change the attributes name. Visual Creation of Views 30

42 Delete not required attributes: select the checkboxes beside the fields and click the Remove Selected Field button. Add a WHERE condition to the UNION operation: click on the Set Condition button (see section 5.4.3). Add derived attributes: click on the New field button to open the expressions editor (see section 5.4.3). We can edit this attribute later by clicking on the icon. Remove the duplicate rows from the output: select the DISTINCT clause checkbox. Order the output by one or more fields: select a field in the ORDER BY drop-down and click on the plus icon beside. Once you have added an ORDER BY field, you can choose to order the output in ascending (ASC) or descending (DESC) order. Select the project for the new view: select a project in the Target Project drop-down. In our example: The new view will be the UNION of the views phone_inc and internet_inc. We will add another association: internet_inc.summary = phone_inc.description. The name of the new view will be incidences. We will eliminate the fields pinc_id, iinc_id, specific_field1, specific_field2 and specific_field3. The target project will be VDPExample. After this, click Ok to create the new view. Then, the schema of the new view will be displayed (see section 5.4.6) Creating Join-Type Views A join-type view executes the relational algebra operation JOIN on a series of input views. To define a join view, open the Toolbox menu and select the Join option. To select the relations on which the join operation is to be executed, drag&drop from the list of views on the left side of the screen. In this part of the example, we will combine data obtained from a Web Service with data from a database. To do this, drag&drop the incidences view and the base view average_monthly_sales (see Figure 16). As input views are added, the schema of the resulting join view is generated automatically. The schemas of the input views are shown on the left of the workspace and the output schema is shown on the right. To add a join condition, select an operator in the drop-down list beside the text Operator and drag&drop one field of the first view into the target field of the other input view. It is possible to define a more complex join condition, by clicking the Add complex condition button. In our example we will select the operator = and drag&drop from the attribute taxid of the incidences view to the attribute taxid of the average_monthly_sales view, thus defining the join condition incidences.taxid=average_monthly_sales.taxid. Visual Creation of Views 31

43 Figure 16 Constructing the incidences and average_monthly_sales join view The join type can also be specified: inner, left outer, right outer or full outer (see FROM Clause in the Advanced VQL Guide [VQL]). A conventional inner join is used in our example. Note: We can only select Join type, Method type and Order if there is at least one Join condition. Otherwise the new view will be a cross join of the two views. The preferences on the join strategy can also be established. This selection may have a significant impact on the performance of queries (see Query Optimization in the Advanced VQL Guide [VQL]). This involves two parameters: Execution method (merge, hash, nested or nested parallel). When the nested_parallel method is selected, we also need to fill in the maximum number of parallel subqueries ( Nested parallel number ) to be run on the second source (see Query Optimization in the Advanced VQL Guide [VQL] for further information on the nested parallel method). Order in which the input relations must be considered. The view at the top will be considered as a first relation if the Ordered option is selected, and as second, if Reverse Order is selected). In our example, no preference will be established for the run strategy, allowing the VDP server to select the one it considers best. The following actions can be carried out on the output view schema (right side of the screen): Change the name of the new view Visual Creation of Views 32

44 Change the attributes name. Delete not required attributes: select the checkboxes beside the fields and click the Remove Selected Field button. Add a WHERE condition to the UNION operation: click on the Set Condition button (see section 5.4.3)). Add derived attributes: click on the New Field button to open the expressions editor (see section 5.4.3). We can edit this attribute later by clicking on the icon. Remove the duplicate rows from the output: select the DISTINCT clause checkbox. Order the output by one or more fields: select a field in the ORDER BY drop-down and click on the plus icon beside. Once you have added an ORDER BY field, you can choose to order the output in ascending (ASC) or descending (DESC) order. Select the project for the new view: select a project in the Target Project drop-down. In our example: The new view is an INNER JOIN of the views incidences and average_monthly_sales. The JOIN condition is incidences.taxid=average_monthly_sales.taxid. The name of the new view will be incidences_sales. We will delete one of the taxid fields of the target view (incidences_sales) The Join type is Inner The target project will be VDPExample. Once the output view has the desired form, press Ok to create it. Click on the list on the left-side panel to access the schema of the new view (see Figure 17). Section contains information about the available options of this dialog. Visual Creation of Views 33

45 Figure 17 Schema of the incidences_sales join view Join Conditions with Similarity Operators Virtual DataPort supports join conditions that use similarity operators. The similarity operator returns a value between 0 and 1 that estimates the similarity between the operands using a certain algorithm. As well as the operands to compare, this operator receives the similarity algorithm to use and a minimum similarity threshold as parameters. Where the similarity between operands reaches the threshold, the condition is true. Otherwise it is false. DataPort includes the operator ~ (Alt-126) to evaluate the similarity between character string-type operands. For this, DataPort includes algorithms based on the edition distance between the texts: ScaledLevenshtein JaroWinkler Jaro Level2Jaro MongeElkan Level2MongeElkan It also includes algorithms based on the appearance of words in both texts: TFIDF Jaccard UnsmoothedJS Besides, it incorporates combinations of some of these algorithms. I.e: JaroWinklerTFIDF. Visual Creation of Views 34

46 If no algorithm is specified, DataPort chooses the one to apply. In that case, only the similarity threshold has to be specified. Example: Consider a variation of the example in which customers are identified by their name instead of their taxid. Unfortunately, the name of the customers in the incidence databases does not exactly match their name in the sales Web Service and, therefore, a join operation with the equality operator will not give good results. In this case, a join can be used with a similarity operator to solve this problem. The incidences example included with the Denodo s distribution can be used to reproduce this situation. Two additional relational tables are included known as internet_inc_cname and phone_inc_cname. These tables are similar to the internet_inc and phone_inc tables used in the example, although they include a customer_name attribute to indicate the name of the customer instead of the taxid attribute. The sales Web Service also includes an operation known as GetAverageMonthlyRevenueCName that returns a compound data element containing the name and the monthly revenue of all the customers. To solve the problem posed by this example, follow the steps below: 1. Create base views for the internet_inc_cname and phone_inc_cname tables (see section 5.3.3). 2. Create a base view for the GetAverageMonthlyRevenueCName operation (see section 5.3.5). 3. Join the base views created for internet_inc_cname and phone_inc_cname (see section 5.4.1). 4. Flatten the compound data element returned by the view created from the GetAverageMonthlyRevenueCName operation. See section for a detailed explanation of the flattening process. 5. Define a join view using the customer_name field between the view obtained as a result of step 3 and the view obtained as a result of step 4. Specify operator ~ as the join operator and set the similarity threshold to Run the view obtained in step 5. The join is made correctly, despite the fact that the customer names in the input views are not exactly the same Creating Projection / Selection-Type Views A Selection-type view executes the relational algebra operations of selection (filtering) and projection (removing fields) on an input view. To define a projection / selection view, open the Toolbox menu, select the Projection / Selection option and select the input view (drag&drop from the list of views or from the Quick Search panel). In our example, the input view will be incidences_sales (see Figure 18). To add a selection condition, press the Set Condition button that will open the conditions editor. This editor has two modes: Simple condition mode: used to create a list of conditions combined through the AND logical operator, where each condition has an attribute of the view as left operand. By clicking the button beside the Conditions label you can add a new condition to the query. For each condition you have to indicate: o One of the attributes of the view. o The operator of the condition: =, <>, contains, etc o The right operand. It can be any expression using constants, view attributes and functions (see Functions for Conditions and Derived Attributes in the Advanced VQL Guide [VQL]). If the selected attribute is of a compound type or of xml type, click on the button beside the text box to open, respectively, the Compound values editor (see section ) or the XML value editor. Specify WHERE expression mode: used to write any condition. This is required, for instance, when you need to use the OR and NOT operators, or when the left operand of the condition is an expression. Auto- Visual Creation of Views 35

47 complete features are available to help constructing syntactically valid expressions. You can also create complex conditions in a purely graphical manner by accessing the Advanced Conditions Editor (see section 12.3). The condition operators available in DataPort are described in detail in the section Comparison Operators in the Advanced VQL Guide [VQL]. The following actions can be carried out on the output view schema (right side of the screen): Change the name of the new view Change the attributes name. Project an atomic attribute that forms part of a Register element. This can be done by right-clicking on the required atomic field and selecting the option Project subfield. Delete not required attributes: select the checkboxes beside the fields and click the Remove Selected Field button. Add derived attributes: click on the New field button to open the expressions editor. In this editor, you can specify the name of the new field and the expression used to compute it. The expression can use constants, view attributes and functions (see Functions for Conditions and Derived Attributes in the Advanced VQL Guide [VQL]). Auto-complete features are available to help constructing syntactically valid expressions. You can also create complex conditions in a purely graphical manner by accessing the Advanced New Field Expression Editor, which is used in a way analogous to the Advanced Conditions Editor (see section 12.3). We can edit this attribute later by clicking on the icon. Remove the duplicate rows from the output: select the DISTINCT clause checkbox. Order the output by one or more fields: select a field in the ORDER BY drop-down and click on the plus icon beside. Once you have added an ORDER BY field, you can choose to order the output in ascending (ASC) or descending (DESC) order. If there is a selection condition ( Set Condition ), select one of the WITH CHECK OPTION options to force every row that is inserted or updated to conform to the definition of this view (see Use of With Check Option in [VQL] for more detail). The available options are: o o WITH CHECK OPTION local: the selection condition of this view is checked when a row is inserted or updated. It does not check the conditions of the lower-level views (the views that participate in the definition of this view). WITH CHECK OPTION cascade: the projection conditions of this view and the lower-level views are checked when a row is inserted or updated. Select the project for the new view: select a project in the Target Project drop-down list. In our example: The input view will be incidences_sales The name of the new view will be pref_clients_inc_sales The projection condition will be revenue>600 to retain only data of those clients with an average monthly volume of sales that exceeds 600 euros. Visual Creation of Views 36

48 The target project will be VDPExample. Figure 18 Constructing the selection view pref_clients_inc_sales Once the schema of the output view has the desired settings, press the Ok button to create the new view. It will appear in the list on the left part of the screen Creating Conditions with the Compound Values Editor When using the Simple condition mode to create a condition, we have two options to create constants of compound types (register or array) 1. Write the value directly in VQL syntax. I.e: { ROW( 'B ' ), ROW( 'B ' ) } See Advanced VQL Guide [VQL] for more details about this syntax. 2. Or, use the Compound values editor. To open it, click on the button beside the text field of the right operand of a condition (only displayed when the selected field is a register or an array) Next, as an example, we will create a new selection view using the Compound values editor to create a compoundtype constant. But first we need to create a new base view: Visual Creation of Views 37

49 1. Using the Web Service datasource sales (see section 5.3.5), create a base view based on the getsumrevenuebytaxids operation. This operation has an input parameter which is an array of strings (the taxid of each client); and returns the sum of the revenue of all specified clients (float-type value). 2. Rename the new base view to RevenueSum. 3. Rename its attributes to clients and totalrevenue. The clients attribute is an array of registers of the type getsumrevenuebyids_in0. Virtual DataPort generates this type automatically when it creates the view. Each element in this array will be a register of the type getsumrevenuebyids_in0_string, which is also automatically generated by DataPort. 4. Rename the element of the clients array to taxid. 5. Click Ok to create the base view. Now we will create a selection view on the RevenueSum base view to obtain the sum of the revenue of the clients with taxids B and B The constant operand of the selection condition will be created using the Compound values editor. The steps to follow are: 1. Select the option Combine Toolbox Projection/Selection. 2. Drag RevenueSum to the work area. 3. Click the Set Condition button to access the selection conditions editor. 4. Click the button to add a condition. 5. Select the field CLIENTS and select the operator =. 6. Click the button to open the Compound value editor. 7. Click twice the button beside Value to create to elements of the type getsumrevenuebytaxids_in0_string. 8. Click on the new elements to expand them and edit the values of each array element. 9. Enter the value B in the field taxid of the first register and B in the same field of the second register (see Figure 19) 10. Click Ok to confirm the values. 11. Click Ok to create the selection view. Visual Creation of Views 38

50 Figure 19 Creating a value of type getsumrevenuebytaxids_in Creating Aggregation-type Views An Aggregation-type view allows executing a GroupBy-type operation on an input view. To define an aggregation view, open the Toolbox menu, select the Aggregation option and drag&drop the input view from the list of views and base views on the left side of the screen (or from the Quick Search Panel) The schema of the input view is displayed on the left of the work area and the current output schema, on the right. Initially the work area will be empty. It is important to remember that an aggregation-type view can have two types of attributes: GroupBy Attributes. Attributes that execute the group-by operation. Attributes resulting from the application of an aggregation expression. Aggregation expressions involve the use of aggregation functions such as COUNT, MIN, AVG, etc. (see Aggregation Functions in [VQL] for the complete list of aggregation functions). To specify the groupby attributes drag&drop the fields over the label Group By Fields in the lower part of the dialog. An attribute of the input view can also be dragged directly to the output view. In this case, the attribute is considered a group-by attribute and will also appear in the output view schema (see Figure 20). Visual Creation of Views 39

51 Figure 20 Creating aggregation view inc_grouped_by_pref_clients The following actions can be carried out on the output view schema (right side of the screen): Change the name of the new view Change the attributes name. Delete not required attributes: select the checkboxes beside the fields and click the Remove Selected Field button. Add aggregation attributes: click on the New aggregation expression to open the aggregation expressions editor. In this editor, you can specify the name of the new attribute and the aggregation expression used to create it. Auto-complete features are available to help create syntactically valid expressions. We can edit this attribute later by clicking on the icon. Add a WHERE condition to the projection / selection operation: click on the Set Condition button (see section 5.4.3). Add derived attributes: click on the New Field button to open the expressions editor (see section 5.4.3). We can edit this attribute later by clicking on the icon. Visual Creation of Views 40

52 Remove the duplicate rows from the output: select the DISTINCT clause checkbox. Order the output by one or more fields: select a field in the ORDER BY drop-down and click on the plus icon beside. Once you have added an ORDER BY field, you can choose to order the output in ascending (ASC) or descending (DESC) order. Select the project for the new view: select a project in the Target Project drop-down. In our example: The input view will be pref_clients_inc_sales The name of the new view will be inc_grouped_by_pref_clients The group-by fields will be taxid and revenue. Add a new aggregation attribute named num_incidences resulting from the aggregation expression count(*) The target project will be VDPExample. Once the output view has the desired settings, press the Ok button to create it. Clicking on it on the list on the left part of the screen will provide access to its schema Flattening of Register-array-type Fields Denodo Virtual DataPort supports the modeling of data types with tree structure using the types register and array (see Advanced VQL Guide [VQL]). In Virtual DataPort, an element of the type array can be considered a sub-view; in this way, an array type always has an associated register type that acts like the schema of the sub-view it is modeling. Sometimes you may wish to flatten a compound field that contains an array of registers. This is particularly frequent when processing XML- and Web-services-type sources. This section describes how this is done. It is important to note that if you wish to directly flatten a register (and not an array of registers), this should be done by creating a new projection/selection view (see section 5.4.3). Going back to the sales Web Service data source, we will now use the operation getaveragemonthlyrevenue. This operation does not have any input parameter and returns the sales data from all the clients as an array of objects, where each object has two properties: taxid and revenue (see Figure 21). Visual Creation of Views 41

53 Figure 21 The operation getaveragemonthlyrevenue from the Web Service sales The name of this base view will be average_revenue_array (see Figure 22). It is difficult to combine information from a view that returns an array. So, we will create a new flattened view with the two attributes taxid and revenue from the array. The name of the new view will be flat_revenue. Visual Creation of Views 42

54 Figure 22 Default base view created for the Web Service operation getaveragemonthlyrevenue To carry out a flattening operation on a view: 1. Select the option Toolbox - Flatten on the menu on the left side 2. Drag&drop the view to be flattened over the workspace. The schema of the input view will appear on the left part of the workspace and on the right part, the schema of the output view (initially both schemas will be equal). 3. Right-click the array-type fields of the input view and select the Flatten array option (see Figure 23). Visual Creation of Views 43

55 Figure 23 Flattening the array of the base view average_revenue_array The output view will then show the flattened schema (see Figure 24). Figure 24 Output schema of the base view average_revenue_array flattened Visual Creation of Views 44

56 In each flattening operation it is possible to select just one array. If there is more than one array, this operation must be repeated for each one. The following actions can be carried out on the output view schema (right side of the screen): Change the name of the new view Change the attributes name. Delete not required attributes: select the checkboxes beside the fields and click the Remove Selected Field button. Add a WHERE condition to the projection / selection operation: click on the Set Condition button (see section 5.4.3). Add derived attributes: click on the New Field button to open the expressions editor (see section 5.4.3). We can edit this attribute later by clicking on the icon. Remove the duplicate rows from the output: select the DISTINCT clause checkbox. Order the output by one or more fields: select a field in the ORDER BY drop-down and click on the plus icon beside. Once you have added an ORDER BY field, you can choose to order the output in ascending (ASC) or descending (DESC) order. Select the project for the new view: select a project in the Target Project drop-down. Finally, pressing the Ok button creates the new view obtained as a result of the flattening operation (see Figure 25). Figure 25 Schema of the flat_revenue view Visual Creation of Views 45

57 5.4.6 Viewing the Schema of a View To access the schema of a view, click on the view on the left side of the screen. In addition to the options available for base views (see section 5.3.4), we also have: Tree View: see section Data Lineage: see section Figure 26 Schema of the view incidences Tree View Mode Using the Tree View button on the view s schema dialog displays the view in tree mode. This visualization mode graphically displays the successive levels of views that have been composed to construct this view. For example, Figure 27 shows the tree of the view incidences_sales. The available options in this dialog are the following: Click on a view to access the page showing its schema. Click on the tree nodes representing combining operations (joins, unions, selections, etc.) to display their main properties. For example, for selection operations, it will display their select condition. Click on a datasource (leaf nodes) to see its properties (only in administration mode) Sensitive information such as passwords will not be displayed. Visual Creation of Views 46

58 Click on Save image to export the Tree View into an image file. Figure 27 Tree view of incidences_sales Data Lineage This dialog displays a diagram of the data sources and views used to form the current view and where the information of each field comes from. On the left side of the dialog, there is a list of all the view's fields. By clicking on one of the fields, all the views and data sources that participate in the creation of this field will be highlighted. For instance, if a field f is obtained by evaluating an expression involving two fields f1 and f2 from different data sources, DataPort will highlight the data sources (and their associated views) providing f1 and f2, and the view where the expression to obtain the value of f is defined. The available options in this dialog are the following: Click on the tree nodes representing combination operations (joins, unions, selections, etc) to display their main properties. View the details of the data sources by clicking on them (only if connected in administrator mode) Click on a view or base view to open the dialog showing its schema. Click on Save image to export the current Data Lineage Tree into an image file. Visual Creation of Views 47

59 Figure 28 Data lineage of incidences_sales Executing Views The Execute button of the view s schema dialog, obtains the tuples of the view. The default query is select * from viewname. By clicking the icon beside the Conditions label you can add AND conditions to the query. For each condition you have to indicate: One of the attributes of the view. The operator of the condition: =, <>, contains, etc The operand. It can be any expression using constants, view attributes and functions (see Functions for Conditions and Derived Attributes in the Advanced VQL Guide [VQL]). If the selected attribute is of a compound type or of xml type, click on the icon beside the text box to open, respectively, the compound values editor (see section ) or the xml editor. It is possible to select the Specify Where expression option to write a more complex query. In the text box for writing the query, you can use the auto-complete features to create the expression more easily. When constructing queries, keep in mind that the query conditions should fulfill the view s mandatory capabilities (see section 5.6.3). The following options are available in this dialog: Do not use cache. If selected, this execution of the query ignores the contents of the cache and obtains the data from the original sources even if the cache has been enabled for this view. If this view is a combination of views, the contents of their caches will also be ignored. Visual Creation of Views 48

60 Do not use swap. If selected, this execution of the query will not swap the intermediate results to disk even if swapping has been enabled. If this view is a combination of views, their intermediate results will not be swapped to disk either. Limit Rows. Limits the number of rows returned by the query execution. Execute with TRACE. If selected, it will enable the View execution trace button of the dialog with the results of the query. Query Plan. Provides a look ahead at the execution plan that DataPort will use to run the query (see section ). Any query execution can be cancelled at any moment. If results are not being shown yet, cancellation can be achieved by pressing the [X] button of the Please Wait dialog which can be seen until the first batch of results is shown. Once the query results are being shown, the execution can be stopped by pressing the Stop button. Figure 29 shows the result of executing the view incidences_sales. The View Execution Trace button allows accessing to the query execution trace (see section ) once the execution has finished. Figure 29 Executing the view incidences_sales If the response to the query execution contains a compound field (registers or arrays of registers), double click on the field to navigate its subcomponents. For example, Figure 30 shows the result of executing the base view average_revenue_array (described in section 5.4.5). This view returns as a result one sole tuple that contains one field of the type array of registers. Each register contains the taxid and the average monthly sales of a client. Double clicking on the field [Array] provides access to its contents (see Figure 31). The Back button can be used in this screen to return to the higher level. Visual Creation of Views 49

61 The data obtained in the execution can be exported to a CSV file by clicking the Save Output button. The available options are: Select the separator character. Include the metadata of the query: the VQL command, number of rows, query plan, etc. Include the columns names in the file. Figure 30 Result comprised of one tuple with a field of the type array of registers Visual Creation of Views 50

62 Figure 31 Content of a field of the type array of registers Execution Trace of a Statement After executing a statement, it is possible to access to a trace of its execution. It is also possible to obtain the execution plan before executing the query. In both cases, the information displayed is the same, except for certain parameters that are only known at run time (e.g. execution time). The statement execution plan is displayed in a tree diagram, where each node represents either an intermediate view involved in the execution of the statement or an access to a data source via a wrapper. If you want to store this diagram into an image file, click on Save image. The most relevant parameters are displayed for each node on the query execution tree. You can copy the value of each parameter by right-clicking on it and then, clicking on Copy or copy the whole trace by clicking on Copy to clipboard. These parameters include: Node type. If the node is a view, it indicates the type of the view (base view, union, join, selection/projection, etc.). If it is an access to a data source (wrapper), this indicates the type of source (JDBC, ODBC, Web Service, etc). Advanced parameters. These provide further details on each node type. For example: o In the case of wrapper-type nodes, the exact sub-queries executed on each data source and the connection data used to access each one. o o For each view-type node, whether the cache and swapping have been used. No delegation cause explains why a query was not delegated to the source database. When a query uses two or more views that relay on the same JDBC or ODBC data source, Virtual DataPort will try to delegate the entire query to the source database. This way, it can obtain the whole result executing a single query. When DataPort cannot delegate the entire process of the query to a source database, it will show the icon beside the view that could not be delegated Visual Creation of Views 51

63 and the 'No delegation cause' parameter will show the reason why the delegation was not possible. For example, the query may use a function that is not supported by the source database, which will force DataPort to post-process the results obtained. The No delegation cause message will help us to rewrite the view so it can be delegated. Execution time. Time spent executing a part of the query. Start time. The exact moment at which node processing began in the execution plan. End of query time. The exact moment at which node processing (and that of all its children) ended in the execution plan. Time obtaining the first tuple of results. Time spent until the node received the first tuple to be processed. Number of tuples processed. Number of tuples processed by the node. Status. This indicates whether the node was correctly executed or whether an error occurred. As an example, Figure 32 shows part of the graphic trace of the execution of the incidences view (created in section 5.4.1). One of the main utilities of the trace function is debugging. In that case, the nodes that caused an error during the execution will be marked red. By clicking on them, more information about the error will be shown. Figure 32 Incidences view execution trace Figure 33 shows the execution trace of the incidences_sales join view in the case where the Web Service data source is not accessible. Visual Creation of Views 52

64 Figure 33 Join view execution trace with errors Using VQL Shell The VQL Shell tool in the VQL Shell tab allows writing and executing VQL statements on the server. It is possible to write one or more statements separated by ;. The statements will be run on pressing the Execute button or on pressing the key combination CTRL+ENTER. The VQL Shell provides the following options: Stop. Stops the execution of the current query. Load. Loads a VQL script from a file. The Verbose Output option will print the result of the execution of each VQL statement included in the script. It is also possible to save these results into a file. View Execution Trace. Displays the execution trace of the last statement. This option is only enabled if the statement contains the TRACE clause at the end (see section ). Limit Rows. Limits the number of rows returned by the query execution. Save output. Export the result of the last executed query to a CSV file. The available options for exporting are: o Select the separator character. o Include the metadata of the query: the VQL command, number of rows, query plan, etc. o Include the columns names in the file. The VQL Shell includes a command log that saves the last VQL statements run. There are two ways of browsing the command log: Visual Creation of Views 53

65 Using the selectable Commands log. Using the keys F1 (back to previous command) and F2 (forward to next command). It is possible to use the menu on the left to speed up writing VQL statements: Click on a view to add its name to the current VQL statement. Right-click a view and the command SELECT * FROM <view name> will appear in the VQL text box. Click with the mouse wheel button and DESC VQL <view name> will appear in the text box Editing / Replacement of a View Definition Sometimes, it is necessary to modify a Virtual DataPort view. Note: If it is necessary to edit an existing view because the underlying data source has changed, use the Source Refresh option (see section ) In some cases, you can modify a view without regenerating the views that use the modified view. This can be done in two ways: Open the schema of the view and click on Edit. If you are editing a base view, you can change its name and modify the name and type of the view s attributes. If you are editing a derived view, you can rename it and change the specific configuration of the view. For instance, in join views, you can change the join condition, the join strategy, etc. Create a new view with the same name as the view to be replaced. This is useful to: o Change the operation used to create the view. For example, to convert a join view into a union view. o Change the data source of a base view. That is, if the data has to be obtained from a different type of source. When a change in a view affects other views, a dialog will be shown listing the changes before they are applied (see section ). If the changes are applied but they are incoherent with the derived views (e.g. on updating a view taking part in a join operation in which any of the join attributes disappears), the icon will be displayed beside the view s name in the Combine menu. If you only need to rename the view, in the left-side menu, right-click on the view and then click on Rename. Visual Creation of Views 54

66 Figure 34 Confirmation of the replacement of a view Source Refresh The Source Refresh option updates a base view, reflecting the changes in the underlying data source. To refresh a base view, follow these steps: 1. Locate the base view in the menu Combine > Projects > <project name> > Base Views of the left part of the screen. 2. Open the base view by clicking on it. 3. Click Edit. 4. Click Source Refresh. If the modifications in the base view affect other views, the Administration Tool will display a dialog with a list of the affected views. Select which views you want to propagate these changes to, by selecting the check boxes that appears right before each view s names (select the header's check box to select all the views in the list) After accepting the changes, the views left in an incorrect state appear in the Combine tab marked with a red exclamation mark Views Affected by Modifications Whenever a change in a view affects other views, this dialog will be displayed. It contains a list of all the views that will be affected by these modifications. Click on one of them to see how that view is related to the modified one. If you have added a new field to the view, you can choose if you want to add this new field to all, some or none of the related views. Visual Creation of Views 55

67 Figure 35 View affected by modification to another view 5.5 OTHER TYPES OF SOURCES This section describes the following actions: Importing data sources of the following types: ODBC, XML, JSON, Web (created using Denodo ITPilot), Aracne, Google Mini, LDAP, delimited text files and CUSTOM sources. Creating base relations for these data sources. Creating a base relation from a SQL query instead of from a view (only for JDBC/ODBC data sources) Configure the data sources properties. When importing data sources of the types XML, JSON or delimited files, it is necessary to specify paths to their files. Section describes how to do so Path Types in Virtual DataPort When we create a data source of the type XML, JSON or delimited text file (DF), we have to specify a path to the data files. Virtual DataPort supports four types of paths to access a file: 1. Local. Path to a file located in the local file system of the Virtual DataPort server. When you click the Browse button, the files listed are the ones located in the system where the server is installed. It does not list the files of the system where the Administration Tool is running. Visual Creation of Views 56

68 Instead of specifying the path to the file, we can use interpolation variables (see section ) if we do not know the path to the file yet. This can be useful if we only know the path to the file at runtime. E.g: we can specify the path The base views created over this data source will have an extra field called report_id and at runtime we will have to provide the value of this field in the WHERE clause of the query. 2. HTTP. Provides access to a file through an HTTP request (GET or POST). The URL can contain interpolation variables (see section ) There are three ways of adding parameters to the body of the POST request (see Figure 36): URL Parameters: if the URL contains parameters, they will be removed from the URL and sent along with the body of the request. XML Body: the XML in the text area will be sent in the body of the request. JSON Body: the JSON in the text area will be sent in the body of the request. Note: You can use interpolation variables (see section ) into the parameters sent to the server. You will be asked for the value of these parameters when this HTTP connection is tested and when a new base view is created over this data source. Figure 36 Http POST request with XML Body The supported authentication methods are: Basic. The credentials are sent in plain text [HTTP-AUTH]. Digest. The credentials are sent encoded [HTTP-AUTH]. Visual Creation of Views 57

69 NTLM. Uses the Microsoft's NT LAN Manager (NTLM) Authentication Protocol [MS-NLMP] to access Microsoft Windows servers. Virtual DataPort supports NTLM v1 and NTLM v2. You can use a specific proxy configuration for this data source or use the Default configuration of the DataPort server (see section 8.1.5). If the HTTP connection is secure (HTTPS) select the Check certificates check box to accept only the certificates that the Java Virtual Machine considers valid. Otherwise, every certificate will be accepted. See the keytool reference [KEYTOOL] for details on how to import an SSL certificate into the Sun Microsystems JVM. 3. Denodo Browser. The file will be obtained using the ITPilot's Denodo Browser, which is capable of executing complex navigation sequences written in ITPilot's NSEQL (Navigation SEQuence Language). See the Denodo ITPilot User Guide [ITPILOT] and the NSEQL Manual [NSEQL] for more information about the Denodo Browser and the NSEQL sequences. 4. FTP / SFTP / FTPS. Provides access to a file available in an FTP server. The parameters required to configure this path are the URL and credentials of the server. If the Test Connection box is selected, the server will check if the route is accessible. If you have specified interpolation variables, you will have to provide their values at this point to be able to test the connection. Besides, in delimited file datasources, there are some additional parameters to specify the charset encoding of the file. If the Autodetect encoding checkbox is selected, DataPort will try to automatically detect the file encoding through a test connection. If the Autodetect encoding checkbox is de-selected, it is possible to choose the desired encoding from the options in the Charset Encoding drop-down list Compressed or Ciphered Data Sources Virtual DataPort can access data files that have been compressed with ZIP; or have been encrypted with the algorithm Password-Based Encryption with MD5 and DES (PBE with MD5 and DES). This encryption method is described in [JCA]. In the new data source dialogs, the available options are: Decompress. Select if the data file is compressed. Decrypt. Select this option if the data file was encrypted with the algorithm PBE with MD5 and DES. In this case, you have to type the password of the file into the Password box Paths Pointing to a Directory A Local path to a data source can point to a single file or to a directory. When creating a base view from a data source that points to a directory, DataPort infers the schema of the new view from the first file in the directory. And it assumes that all the other files have the same schema. Note: for XML data sources, if a Validation file has been provided, all files in the directory have to match that Schema or DTD Paths with Interpolation Variables Paths can be parameterized depending on the query made by using interpolation variables (see Execution context of a query and interpolation strings in the Advanced VQL Guide [VQL]). For instance, the HTTP path: Visual Creation of Views 58

70 uses a variable called DPT_NAME that allows deciding at query runtime the exact file accessed by the path. For instance, if the query specifies as selection condition DPT_NAME = it, then the accessed URL would be: If the path specified includes interpolation variables, an intermediate step (see Figure 37) will be required during the base relations creation process, in which Virtual DataPort will request valid values for them. This is necessary so that DataPort can access a data file and obtain the metadata for creating the base relation. Figure 37 Providing input values to interpolation variables Each interpolation variable must be associated to a generated base view attribute. If the specified data file includes a field with the same name as the variable, then the variable will be associated with the base view attribute corresponding to this field. Where the SQL query specified returns no field with the same name as the variable, DataPort will add a new attribute to the base relation with the name of the variable. NOTE: When interpolation variables are used in the path description, the \ character escapes the special characters used to indicate variable names {, } ). In Windows systems, this character is also used to specify file paths, causing some ambiguous situations. For example, the system could interpret that the path c:\tmp\@{var} references a file named tmp@{var} in the c:\ directory, because the second \, due to its appearance just is considered an escape character. However, the c:\tmp\\@{var} path would reference a file which name is the value of the var variable and it is found at c:\tmp. Visual Creation of Views 59

71 5.5.2 ODBC Sources Virtual DataPort provides access to ODBC sources. Open the Connect Projects ProjectName Relational DB tab and click the option New beside ODBC. The new ODBC datasource dialog will be displayed (see Figure 38). The following data are requested in this screen: Name. Name of the new data source. Database Adapter. Adapter used to connect to the DBMS. If the Choose automatically option is selected (recommended), DataPort will discover the most suitable adapter by connecting to the source. Connection type. There are two types of ODBC connections: o DSN (Data Source Name). Use one of the existing DSNs of the system where Virtual DataPort server is installed. o Direct. Access to a file stored in the local file system of the DataPort server, using one of the ODBC drivers registered on the DataPort server. This is useful, for instance, to access Excel files by using ODBC. The path can use interpolation variables (see section ). Login. User login to access the external database. Password. User password to access the external database. Choose automatically. If the Choose automatically option is marked (recommended), the system will connect to the ODBC data source to discover the most suitable adapter. Test Connection. Virtual DataPort will connect to the source to verify it can be accessed. Click on Connections Pool Configuration to configure various parameters of the connections pool that Virtual DataPort uses to optimize access to external databases. The available parameters are the same as in the JDBC pool (see section 5.3.1). Click on DSN Properties Configuration to add properties (field/value pairs) to the connection URI. Visual Creation of Views 60

72 Figure 38 Importing from an ODBC datasource Once the ODBC source has been imported, the process for creating base relations is the same as that for JDBC (see section 5.3.3). The Excel ODBC driver provided by Microsoft Windows has some limitations: - It cannot DELETE data from Excel spreadsheets. It can only query them and append more data. - The Excel driver of MS Excel 2003 or earlier, can execute only one query simultaneously. This can be a problem if we are combining data from the same Excel file. The newer versions of the Excel driver do not have this limitation Multidimensional Database Sources Virtual DataPort can obtain data from multidimensional databases such as SAP BW3, SAP BI 7 and Mondrian. Note: before creating a multidimensional database source that connects to SAP BW or SAP BI, we have to install the SAP Business Intelligence JDK in the system where Virtual DataPort is running (see appendix ) To create a new data source to retrieve data from a multidimensional database, open the Connect Projects ProjectName tab and click on New beside Multidimensional DB (see Figure 39). The following data are requested in this dialog: Name. Name of the new data source. Database Adapter. Select the version of the database. XMLA URI. The XMLA URI to access the database. E.g: Visual Creation of Views 61

73 System Name. Name of the instance of the database (only for the SAP BW and SAP BI adapters). Language. Language of the connection established with the server (only for the SAP BW and SAP BI adapters). User Name and Password. Credentials to access the database. Figure 39 Creating a data source to a multidimensional database Click on Ok to create the new data source and then, on Create base view from MDX query to create a base view (see Figure 40) Visual Creation of Views 62

74 Figure 40 Create a base view from an MDX query (multidimensional data source) The following data are requested in this dialog: The name of the new view. An MDX query. Appendix 12.2 explains how the result of an MDX query is mapped to a relational structure XML Sources To create a new data source to access an XML document, open the Connect Projects ProjectName tab and click on New beside XML. The new XML data source dialog will be displayed (see Figure 41) The following data are requested in this dialog: Name. Name of the new XML data source. Data Route. Path to the XML file that contains the required data. This path can be parameterized by using interpolation variables (see section ). Validation type. If selected, the structure of the input XML file will be obtained from a Schema or a DTD. If None is selected, Virtual DataPort will analyze the XML document to infer its schema. Validate route. Path to the Schema or DTD of the input XML document. This file will be used to obtain the structure of the input XML file. Visual Creation of Views 63

75 Validate data. If selected, Virtual DataPort will validate the input XML file every time the data source is accessed. The path formats available are described in detail in section Figure 41 Importing from an XML data source Click Ok to create the new data source. Then the XML source description dialog will be displayed (see Figure 43). On clicking the Create Base View button, an Xpath [XPATH] expression pointing to a node in the XML document can be provided (see Figure 42). This is useful to access only to a portion of the document (this may result in a much faster access to the document). In this case, Virtual DataPort will generate a tuple for each node matching with the path. The subelements of these nodes will be considered as fields in the tuples extracted. If, on the Xpath specification screen, the options RSS or ATOM are marked, the system preconfigures an XPath that is normally suitable for the documents of these types. For example, the preconfigured path for RSS is /rss/channel/item. This path will lead to a tuple being generated in DataPort for each RSS feed item. The subelements of the items will appear as the tuples attributes. Visual Creation of Views 64

76 Figure 42 RSS Xpath NOTE: For performance reasons, the XPath element should be indicated when we are not interested in the full document data. Although it is possible to obtain the same data by creating a base view to access the full XML document and then create projection/selection and flattening-type views to filter the data, this will result in a less efficient access. After configuring the path, the schema of a base relation associated with the source is automatically displayed (except if the path to the data file includes interpolation variables, in which case there will be an intermediate step. See section 5.5.1). The name of the base relation and the name and type of each of its attributes may be changed at this stage. The target project of the new view can be specified with the drop-down Target Project. Visual Creation of Views 65

77 Figure 43 Description of an XML source By clicking the new base view, we can access the dialog, which displays its schema (see section 5.3.4) Once the base relation has been created, the Edit button allows accessing the view edition wizard, where we can modify the name and type of the base relation attributes JSON Data Sources To access a JSON data source, open the Connect Projects ProjectName tab and click the New button beside JSON. The new JSON datasource dialog will be displayed (see Figure 44) The following data are requested in this dialog: Name. Name of the new JSON data source. Data route. Path to the JSON file that contains the data. This path can be parameterized according to the query made using interpolation variables (see section ). The formats of the available paths and how to configure the use of parameters in them are described in detail in section The most common path type in this case is HTTP. By clicking on OK, the description screen for the new JSON data source is displayed. By clicking on Create base view, a new base view will be created and its schema will be displayed (except if the path to the data file includes interpolation variables, in which case there will be an intermediate step. See section ). The name of the base view and the name and type of its attributes can be changed. The target project of the new base view can be specified with the drop-down Target Project. Visual Creation of Views 66

78 Figure 44 Importing from a JSON datasource Web Sources (WWW) Web data sources use wrappers created with the Denodo ITPilot Wrapper Generator Tool [ITPILOT]. Note: We can only use these sources if Denodo ITPilot is installed in the same path as Virtual DataPort. To create a new Web data source, open the Connect Projects ProjectName tab and click the option New beside WWW (see Figure 45). This dialog requests a VQL file generated with the ITPilot s Wrapper Generator Tool (see [GENITP]). We can also deploy Web wrappers into Virtual DataPort straight from the Wrapper Generator Tool. Visual Creation of Views 67

79 Figure 45 Importing a WWW wrapper Click Ok to create the new Web source. Clicking on the new data source in the menu of the left side of the screen will open a dialog with the schema of the new data source. The name of the base relation and the type of its attributes can be changed. The target project of the new base view can be specified with the drop-down Target Project. After doing this changes, click Ok to create a base view that will access this new data source Delimited File Sources This type of source access text files in CSV (Comma Separated Values) format or similar. Complex regular expressions can also be used to extract the desired data from other kinds of text files (e.g. application log files). Visual Creation of Views 68

80 Figure 46 Importing from a new delimited file source To create a new data source to access a delimited file (DF), open the Connect Projects ProjectName tab and click the option New beside Delimited Files (see Figure 46) The following data are requested in this dialog: Name. Name of the new data source. Data route. Path to the delimited file. The formats of the available paths are described in detail in section The path can be parameterized according to the query made using interpolation variables (see section ). NOTE: If the path points to a directory, Virtual DataPort will consider the data from all the files in it (all the files should follow the same data format). Column delimiter. Character string to be used as a separator between fields of the same tuple. For instance, the delimiter can be a semi-colon (;), coma (,), dash (-) or similar. Alternatively, a Tuple Pattern can be used (see below). Tuple pattern. Regular expression that specifies the format of the tuples that will be extracted from the delimited file. This regular expression has to match the whole line of the file, not only the part that wants to capture. The syntax of these regular expressions is the one defined by the JAVA language [JREG]. The fields of the views created over this data source will be the capturing groups of the regular expression. Note: the tuple pattern can contain interpolation variables (see section ). Example 1 of tuple pattern: Suppose a file that contains product information in the following format (note that the discount attribute is optional): Visual Creation of Views 69

81 product_name=acme Laptop Computer;price=1500 euro;discount=50 product_name=acme Desktop Computer;price=1000 dollar Figure 47 Example 1 of tuple pattern: contents of the delimited file The following pattern can be used to extract from each row, the name of the product, its price and currency, and its discount. For the tuples without a discount value, the value of this cell will be NULL. product_name=(.+);price=([0-9]+)\s(.+);(?:discount=(.+))? Figure 48 Example 1 of tuple pattern: regular expression to extract the contents of the file Example 2 of tuple pattern: Suppose we want to extract the name of the files (not directories), its date and its size, from the output of the Windows command DIR: 11/07/ :10 <DIR>.dbvis 09/18/ :09 <DIR>.eclipse 01/19/ :55 <DIR>.gimp /10/ : ITLMRegistry 03/26/ : keystore 05/18/ :56 <DIR>.m2 02/02/ :23 <DIR>.maven 03/26/ :01 <DIR>.netbeans 09/23/ :04 36.org.eclipse.epp.usagedata.recording.userId 02/02/ :20 <DIR>.smc_cache 06/15/ :59 <DIR>.ssh 10/14/ :26 <DIR>.thumbnails 02/15/ :06 0.Xauthority 01/16/ : ant.install.log 02/11/ :29 <DIR> Application Data 07/16/ : build.properties 02/18/ :19 <DIR> Contacts 01/14/ : default-soapui-workspace.xml 09/16/ : denodo-v46-update jar 02/07/ :44 <DIR> Desktop 04/01/ :11 <DIR> Favourites 08/22/ :50 <DIR> Start Menu 01/27/ :18 <DIR> My Documents 02/12/ : osdadmin.ini 02/12/ : osdadmin.ini.bak 01/14/ : soapui-settings.xml 02/09/ : temp.txt 04/19/ : test.cert 02/10/ :22 <DIR> Tracing 03/05/ :41 0 vdpws.log 04/17/ :49 <DIR> workspace Figure 49 Example 2 of tuple pattern: output of the Windows command DIR The regular expression we have to put in the Tuple pattern field is: (\d{2}?/\d{2}?/\d{4}?)\s+?\d{2}:\d{2}.*?(\d[^ ]*)\s(.*) Figure 50 Example 2 of tuple pattern: regular expression to extract the output of the command DIR The base views created with this tuple pattern will have three fields: the date of the file, its size and its name. Visual Creation of Views 70

82 End of line delimiter. Character string to be used to mark the end of a tuple. The default value is \n (carriage return). Begin of Data Zone Delimiter. JAVA regular expression identifying the position in the file where the system must start searching for tuples (or searching for the header if the header option was checked). If empty, the search will start at the beginning of the file. Include Begin Delimiter as Data. If selected, the text matching the Begin of Data Zone Delimiter expression will be included in the search space. End of Data Zone Delimiter. JAVA regular expression identifying the position of the file where the system must stop searching for tuples. If empty, the search will continue until the end of the file. Include End Delimiter As Data. If selected, the text matching the End of Data Zone Delimiter expression will be included in the search space. Header. If selected, DataPort considers that the first line of the data region contains the names of the fields in this file. These names will be the fields names of the base views created from this data source. Header Pattern. Regular expression used to extract the name of the fields that form the header. This only needs to be specified if the header has a different structure than the tuples with data. This option can only be used when the Header check box is selected. Click Ok to create the new data source. The delimited files source description dialog will be displayed (see Figure 51). Pressing Create Base View creates a base view associated with the new data source (except if the path to the data file includes interpolation variables, in which case there will be an intermediate step. See section ). The name of the base relation and the name and type of its attributes can be changed at this stage. The target project of the new base view can be specified with the drop-down Target Project. Visual Creation of Views 71

83 Figure 51 Description of a delimited-file-type source By clicking on a base relation we have created, we can access the page that displays its schema in Virtual DataPort Once the base relation has been created, the Edit button allows accessing the view edition wizard, where we can modify the name and type of the base relation attributes Paths Using Date Ranges When we need access to files that use a date-based naming convention (as is typical in log files), the DateRange function can be used to consider only the files between a given start date and a given end date. The function admits the following parameters. - Date format to be used in the queries. - Start date specified in the above format. - End date specified in the above format. - Date format used by the naming convention of the files. Example: Let us suppose the /tmp path contains the log files daily generated by a certain application. The name of these files has the format conflogyyyymmdd (e.g. conflog ). To add the new delimited files data source, you could specify the path: /tmp/conflog^daterange("yyyy/mm/dd",@startdate,@enddate,"yyyymmdd") This way, the base views created from this datasource will include two attributes called startdate and enddate that can be used to specify that a certain query should only consider the files in the specified date range. For instance, the following query: Visual Creation of Views 72

84 select * from myview where startdate='2008/11/01' and enddate='2008/11/19' will obtain data from the log files generated from 01/November/2008 to 19/November/2008. Notice that DateRange can be also used with paths pointing to directories. For instance, let us suppose that the application in the above example stores the log files of each day in a separate directory using the format yyyymmdd. Then, we could simply specify the path: /tmp/^daterange("yyyy/mm/dd",@startdate,@enddate,"yyyymmdd") In this case, the former query would examine all the files from every directory in the specified date range Google Enterprise Search / Mini Sources Virtual DataPort can use Web search engines built using the Google Enterprise Search / Google Mini software, as data sources [GMINI]. To create a new data source representing a Google Enterprise Search / Google Mini search engine, open the Connect Projects ProjectName Search Engines tab and click on New beside Google Mini. The new Google Search datasource dialog will be displayed (see Figure 52) The following data are requested in this dialog: Name. Name of the new data source. Host name. Name of the machine in which the search engine is accessible. This may be a remote machine accessible on the Internet (e.g. search.acme.com ). Port. Port on which the Google Mini service is run (usually 80). Proxy Configuration. If the connection is established through a proxy, the host name and port where the Proxy is executed must be provided. If it is an authenticated proxy, the user identifier and a valid password must be also specified. It is possible to use the default http proxy configuration (see section 5.1.5) by clicking the option Default. Visual Creation of Views 73

85 Figure 52 Importing a new source of type Google Mini Click Ok to create the new data source and see the Google data source description screen (see Figure 53). This dialog lists the fields available in the search engine (these fields are fixed for all Google Enterprise Search / Mini search engines). Visual Creation of Views 74

86 Figure 53 Description of the Google Mini source Clicking Create Base View creates a base view associated with the new data source (see Figure 54). Visual Creation of Views 75

87 Figure 54 Base View Creation in Google Enterprise Search / Mini Datasources A value can be specified for the following parameters at the top of the screen: Site Collections. Required. It specifies the collections within the Google Enterprise Search / Mini server on which to make the search. The collections are created by the Google server administrator. Its name is upper/lower case-sensitive. It is possible to specify several collections separated by commas. In this case, the search will be made on all of them. The collection that is being searched by a particular search form can normally be obtained by examining the value of the site parameter of its invocations. Client. Optional. It identifies the client making the queries. Google Enterprise Search / Mini servers can be configured to behave differently depending on the client. Languages. Optional. If specified, only documents in that language will be returned. The language must be one of the listed in the Google Enterprise Search documentation [GSEARCHLANG]. Number of Key Match. Optional. Google Enterprise Search / Mini allows the administrator to manually set a priority of the pages. This parameter receives an integer between 0 and 5, where 5 is the top priority. If this value is established, the searches made on this base relation will only return the pages with the priority specified or over. The value of these parameters can be modified later, on the advanced configuration dialog of the base view (see section 5.6.7). The name and type of the fields can be modified. The contents of each one of the attributes is: TITLE. Title generated by Google server for the document. SUMMARY. Summary generated by Google server for the document. URL. Document URL. Visual Creation of Views 76

88 MIMETYPE. MIME type of the document. RATING. Priority assigned by the Google server administrator for the document. This may take values of between 0 and 5, where 5 is the maximum priority. MAXDOCS. Attribute added by DataPort to restrict the maximum number of results returned by a search. METAS. Attribute of Array of records type that contains the metatags for the document. Each record has two fields to indicate the name of the metatag (metakey) and its value (metavalue). CONTENT. Contents of the document. This is the field normally used for searches. SITE. If specified, Google server will only return the documents that belong to that site / collection (e.g. acme.com ). FILETYPE. Extension of the document file. The target project of the new base view can be specified with the drop-down Target Project. The base view is created just by clicking Ok. Once the schema has been accepted, the new base relation may be viewed on the left of the screen. The most common way of querying the base relations built from Google Mini sources is using the CONTAINS operator, which runs complex Boolean searches on indexed textual data (see Support for the CONTAINS operator of each source type in Advanced VQL Guide [VQL]) Aracne Sources Virtual DataPort can create data sources that use search indexes built using Denodo Aracne [ARCN]. To create a new data source representing an Aracne index, open the Connect Projects ProjectName Search Engines tab and click on New beside Aracne. The new Aracne datasource dialog will be displayed (see Figure 55). The following data are requested on this dialog: Name. Name of the new data source. Host name. Host where Aracne Index/Search Engine is running. Port. Port where Aracne Index/Search Engine is listening to connections. The default port is Login and Password. Credentials to access Aracne Index. Visual Creation of Views 77

89 Figure 55 Creating an Aracne data source Click Ok to create the new data source. The Tool will display the schema of the new Aracne data source (see Figure 56). The dialog shows the indexes contained in the Aracne server and the fields included in each one. Visual Creation of Views 78

90 Figure 56 Description of the Aracne source Click on Create Base View to create a new base view associated with the index selected (see Figure 57). The name of the base view and the name and type of its attributes can be changed at this stage. Non-required attributes can be deleted. The base views pregenerated from Aracne indexes will have an attribute of the appropriate type for each field in the Aracne Index. Usually, Aracne indexes are used to index the data extracted by Denodo Scheduler jobs and their schema depends on the type of executed job (see the Denodo Scheduler Administration Guide [SCHED] for further details). Figure 56 shows the fields in an index created by the documents exported by a Scheduler job of type ARN. Visual Creation of Views 79

91 Figure 57 Creating base views in Aracne sources Alongside each attribute of the base view, there is the Create Main Terms option (see subsection ). The target project of the new base view can be specified with the drop-down Target Project. The base view is created by clicking Ok. If the user wants to create a base view and then import another one from the same data source, he/she can click OK and import new view, which will create the view and will go back to the screen where the Aracne server s indexes are listed. Once the schema has been accepted, the new base view may be viewed on the left of the screen. The most common way of querying the base relations built from Aracne sources is using the CONTAINS operator, which runs complex Boolean searches on indexed textual data (see Support for the CONTAINS operator of each source type in Advanced VQL Guide [VQL]) Accessing the Most Relevant Terms of a Document Denodo Aracne is capable of automatically generating the most relevant words of a document or one of its fields, according to the TFIDF (Term Frequency Inverse Document Frequency) relevance measurement. These terms can be accessed via Virtual DataPort, as described in this section. The most relevant terms are accessed as new attributes in the DataPort base view. To create a new attribute in the base view containing the most relevant terms of the value of a certain document field, click on the Create main terms option alongside the required field. For example, if you want to add a new attribute containing the most relevant terms of the SEARCHABLECONTENT index field, click on Create main terms alongside it. Figure 58 shows this new attribute (known as SEARCHABLECONTENT_MAIN_TERM). Visual Creation of Views 80

92 Figure 58 Adding an attribute with the most relevant terms of the SEARCHABLECONTENT field The new attribute SEARCHABLECONTENT_MAIN_TERM will contain an array of records. Each record in the array contains two fields: The relevant term. This field takes by default the name of the index field, adding the suffix _TERM (in this case, therefore, the name will be SEARCHABLECONTENT_TERM). However, as can be seen in the figure, the name of the field can be changed. Its position in the list of the most relevant. This field takes by default the name of the index field, adding the suffix _SCORE (in this case, therefore, the name will be SEARCHABLECONTENT_SCORE). However, as can be seen in the figure, the name of the field can be changed. The most relevant term takes position 1. In the new attribute creation process, it is possible to specify two parameters: Number of main terms. Maximum number of relevant terms to be included for each document. Filter main terms words. List of usual words (separated by commas) that must not appear among the most relevant terms for this field. If the list of relevant terms generated by Aracne includes any of those, they will be removed from the list. It is important to note that only usual words specific to the application must be specified. The usual words in the language used such as articles, pronouns, etc. (commonly known as stopwords ) are already eliminated by Denodo Aracne. The list of usual words may be contained in a file. The file specified must be a text file, where the words will be separated by commas. Furthermore, at the bottom of the base view creation screen it is displayed the General Words to Filter Main Terms parameter that indicates a list of usual words common to all fields in the base view. Once again, you do not have to worry about specifying usual words in the language used such as articles, pronouns, etc. (commonly known as stopwords ), as they are already eliminated by Denodo Aracne. Visual Creation of Views 81

93 The value of these parameters can be subsequently modified on the advanced configuration screen of the base view created (see section 5.6.7) LDAP Sources The LDAP-type data sources allow registering an LDAP server in Virtual DataPort. LDAP-type data sources can be used for two purposes: To extract data stored in the LDAP server. To delegate authentication tasks to an LDAP server: o Authentication of users of Virtual DataPort servers (see section for further details on the creation of users in DataPort) o Authentication of users of Web Services published by the Virtual DataPort server (see section 6) To create a new data source representing an LDAP server, open the Connect Projects ProjectName tab and click the option New beside LDAP. The new LDAP datasource dialog will be displayed (see Figure 59). The following data are requested on this screen: Name. Name of the new data source. Server URI. Path to the LDAP server. For example: ldap://acmehost:389 Login and Password. Credentials to access the LDAP server. Select Use Paging if the LDAP server has a limit on the number of results per query. If selected, Virtual DataPort will do paged searches to obtain all the results of the queries. Select Test connection to test the connection with the LDAP server before creating the data source. Visual Creation of Views 82

94 Figure 59 Creating an LDAP data source After clicking Ok, the Tool will display a dialog containing a list of the different object classes that the LDAP server contains (see Figure 60) Visual Creation of Views 83

95 Figure 60 New LDAP data source Select the desired object classes and click on the Create base view button. Then, Virtual DataPort will create a new base view with the fields of the selected classes and display it to the user. If required, the name of the base view and the name and type of its attributes can be changed at this stage. The target project of the new view can be selected with the box Target Project. Click on Ok to create the base view. Optionally, before clicking on the Create base view button, it can be indicated whether the query should: recursively consider all the nodes in the subtree hanging from the node pointed by the datasource path, or it should only consider the direct children of the node. Once the new data source has been created, it is also available to create new users (see section 9.3.3). It is also possible to create a base view from an expression which is delegated to the LDAP server. For this, you must click on the Create base view from expression button, and the administration tool will show a form to introduce the expression (see Figure 61). This expression can have interpolation variables (see section ). If interpolation variables are used, then the user will be asked for example values for the variables to perform a query against the LDAP server. Using this query, the system will obtain the object classes that can be reached using the expression. The user can then select the object classes to be used to generate the base view schema (see Figure 62). Optionally, it can be specified whether the query to perform against the LDAP server should be recursive or not. The interpretation of this option is the same as in the formerly explained case of creating the base view without specifying an expression. Visual Creation of Views 84

96 Figure 61 New LDAP base view from expression Visual Creation of Views 85

97 Figure 62 Accessible objects of an LDAP base view created from an expression BAPI Sources Virtual DataPort can invoke SAP BAPIs (Business Application Programming Interfaces) to obtain data stored in SAP ERP and other SAP applications. Important: before creating any BAPI data source, we have to install the SAP Java Connector 3 in the system where the Virtual DataPort server is running (see Appendix ) To create a new BAPI data source, open the Connect Projects ProjectName tab and click on New beside BAPI. The new BAPI data source dialog will be displayed (see Figure 63) Visual Creation of Views 86

98 The following data are requested in this dialog: Name. Name of the new data source. System Name. The SAP system ID of SAP ERP. Host. Host where SAP ERP is running Client ID. Client id. Figure 63 Creating a new BAPI data source System Number. The two-digit number to differentiate two SAP ERP running on the same host. User Name and Password. Credentials to access SAP ERP. Click on Ok to create the new data source. Then, click on Create base view from BAPI to create a base view (see Figure 64) Visual Creation of Views 87

99 Figure 64 Creating a new BAPI (SAP ERP) data source Fill in the name of the BAPI and click on Ok to display the schema of the result of the BAPI invocation. We can change the name of the new view and its fields and after that, click Ok to create the view CUSTOM Sources CUSTOM-type data sources (also called MY data sources) are ad-hoc wrappers that have been specifically developed to access a specific source and can be used in Virtual DataPort. See the VQL Advanced Guide to learn to develop CUSTOM wrappers. In the path $DENODO_HOME/samples/vdp/customWrappers there are examples of CUSTOM wrappers. The README file in this path contains instructions on how to compile, install and use them. The screenshots displayed in this section use the sample CUSTOM wrapper com.denodo.vdp.demo.wrapper.custom.metapayrollwrapper. Visual Creation of Views 88

100 Figure 65 Importing a CUSTOM wrapper To create a new data source of this type open the Connect Projects ProjectName tab and click the option New beside Custom. The new Custom datasource dialog will be displayed (see Figure 65). The following data are requested in this screen: Name. Name of the new data source. Class name. Full name of the JAVA class that implements the CUSTOM-type wrapper. Classpath. If the.jar library was not imported to the server using the Jar Management option (see section 8.3), you have to indicate its location by clicking the Browser button and selecting it. Otherwise it can be left empty. Select Jars. If the class belongs to an imported Jar file, select this checkbox and select the appropriate Jar. Note: to select two or more Jars, hold the key Ctrl and click on the required Jars. This may be necessary if the custom wrapper relies on external jars that have also been imported into Virtual DataPort. Where the examples of CUSTOM wrappers included with DataPort have been compiled and installed, follow the instructions and click on ok to create the new data source (see Figure 66). Visual Creation of Views 89

101 Figure 66 New CUSTOM data source Click on the Create Base View button to create a base view associated with the new data source. The CUSTOM wrapper used to implement access to the source can accept input parameters (see Creation of Custom Wrappers in the Advanced VQL Guide [VQL]). In Figure 67, the user has to fill in the values of the following parameters: ENTERPRISE = enterprise1 YEAR = 2007 Visual Creation of Views 90

102 Figure 67 Creating a base view from a CUSTOM data source By clicking on Ok, the schema of a base relation associated with the data source is created (see Figure 68). Visual Creation of Views 91

103 Figure 68 New base view from a CUSTOM data source Select the project assigned to new base view in the drop-down Target Project. By clicking on the base view, we can access the page which displays its schema. Once the base relation has been created, click on the Edit button to access to the view edition screen, which allows renaming attributes or changing their types JDBC/ODBC sources through SQL Query Sometimes, it may be necessary to create a JDBC/ODBC base view using an arbitrary SQL query, instead of using a table of the database as a template. This can be useful in two scenarios: When we need to execute a database stored procedure or a query that uses a stored procedure, to obtain the results. When it is faster to create a base view specifying an SQL query, than importing all the base tables and creating the equivalent combination view. To create a SQL Query-type base relation, use the Create Base View from Query button on the JDBC or ODBC data source detail screen. You will access the screen shown in Figure 69. Visual Creation of Views 92

104 Figure 69 Creating a base view using an SQL query Once the name for the base relation and the required SQL query have been specified, pressing ok creates the schema of the new base relation. Its schema can be edited as usual. The SQL query used to create the base view may also use interpolation variables (see section ), which allow the SQL query made on the source to be parameterized according to the specified query conditions. In this case, after pressing Ok, the user has to fill in the wizard with the values of the variables used (see Figure 70). This is necessary so that DataPort can execute the query to obtain the metadata required to create the base relation. Each interpolation variable must be related with an attribute belonging to the generated base relation, so that the variable can obtain its value at run time. If the SQL query used to access the data source returns any field with the same name as the variable, then the variable will be associated to the corresponding base relation attribute. In case the SQL query does not return any field with the same name as the variable, DataPort will add a new attribute to the base relation, with the same name. If the SQL query invokes a PL/SQL function using an interpolation variable, you need to assign an alias to the return value. For example: SELECT function_test(@var1) AS value FROM internet_inc Visual Creation of Views 93

105 Figure 70 Editing the value of the interpolation variables Using the WHEREEXPRESSION Variable As we commented in the previous section, the SQL Query used to create the base relation can contain interpolation variables (see section and the VQL Advanced Guide [VQL]]), to parameterize the SQL Query executed. There is a predefined interpolation variable called WHEREEXPRESSION that simplifies the process of creating base relations by using the SQL Query method. Besides, the use of WHEREEXPRESSION can also help optimize the execution. Specifically, if a join view is using the NESTED execution method (see the Query Optimization section in the VQL Advanced Guide [VQL]), and the second view in the join operation is a SQL Query view, it is highly advisable to create this second view using the WHEREEXPRESSION variable. This allows DataPort to apply optimization techniques that would not be possible otherwise. The WHEREEXPRESSION variable can be used in the SQL query as substitute of all or part of the WHERE clause in the query. At run time, DataPort will substitute the variable by a valid query condition built from the query conditions received by the base relation. For example, let s suppose that we create a base relation called VIEW1 from the following SQL query: SELECT StorProc(FIELD1), FIELD2, FIELD3, FIELD4 AS ALIAS4 FROM TABLE1 Notice that the query uses a stored procedure in the SELECT clause, so it is necessary to create the base view by using a SQL query. Visual Creation of Views 94

106 After clicking Ok, the user will have to provide the values for the variables used, so DataPort can execute a query to obtain the metadata required for creating the base view (see Figure 70). In this example, any valid query condition could be used as value for WHEREEXPRESSION like, for instance: FIELD2= f2 AND FIELD4= f4 Once DataPort has obtained the required metadata by executing the sample query, it will show the list of base view attributes that will be generated (see Figure 71). The user may specify information about any aliases used in the query SELECT clause. For instance, in the previously commented query, the ALIAS4 alias is defined, which is associated to the FIELD4 field. Therefore, the user must specify the FIELD4 value for the attribute named ALIAS4. After providing the alias information, the base view will be created. Figure 71 Specifying aliases In our example, let s consider what would happen if, once the VIEW1 view has been created, the following VQL query is executed (NOTE: in the example, it is assumed that the user has not modified the attribute names when creating the base relation, and, therefore, they match the ones specified in the SQL query used to create the base relation): SELECT * FROM VIEW1 WHERE FIELD2= f2 AND ALIAS4= f4 In this case, DataPort will substitute the WHEREEXPRESSION variable at run time, by the value required to execute the equivalent query on the original database. In this case: SELECT StorProc(FIELD1)AS ALIAS1, FIELD2, FIELD3, FIELD4 AS ALIAS4 FROM TABLE1 WHERE FIELD2= f2 AND FIELD4= f4 Visual Creation of Views 95

107 Data Source Configuration Properties Data Source Configuration properties allow specifying certain characteristics of the underlying data sources such as the operations they support. Knowing the capacities of each data source is important for optimization reasons since it allows Virtual DataPort to delegate to the data source as much processing as possible to optimize response times and minimize traffic through the network. NOTE: Typically, users do not need to edit this information since DataPort automatically uses suitable configurations for most common data sources. The properties of each data source can be configured by selecting the required data source and pressing the Edit button before clicking on the Source Configuration link. This link only appears in data sources with configurable properties. The configurable properties are as follows: NOTE: Virtual DataPort has default values for some specific relational databases (MySQL, Oracle, Postgres, etc) that vary in relation to the default values mentioned below. - Delegate All Operators (DS: JDBC, ODBC). This indicates whether the source allows for all operators to be delegated. The value is false by default. - Delegate Array Literal (DS: JDBC, ODBC). This indicates whether the source allows for array-type compound constants to be delegated. The value is true by default for JDBC and ODBC sources. - Delegate Compound Field Projection (DS: JDBC, ODBC). This indicates whether the source allows projections on compound fields to be delegated. The value is true by default for JDBC and ODBC sources. - Delegate GROUP BY (DS: JDBC, ODBC). This indicates whether the source allows the GROUP BY clause to be delegated. The value is true by default for JDBC and ODBC sources. - Delegate HAVING clause (DS: JDBC, ODBC). This indicates whether the source allows the HAVING clause to be delegated. The value is true by default for JDBC and ODBC sources. - Delegate Inner Join (DS: JDBC, ODBC). This indicates whether the source allows for the Inner Join operator to be delegated. The value is true by default for JDBC and ODBC sources. - Delegate Join (DS: JDBC, ODBC). This indicates whether the source allows for the Join operator to be delegated. The value is true by default for JDBC and ODBC sources. - Delegate Left Function (DS: JDBC, ODBC). This indicates whether the source allows for conditions with functions on the left part to be delegated. The value is true by default for JDBC and ODBC sources. - Delegate Left Literal (DS: JDBC, ODBC). This indicates whether the source allows for conditions with constants on the left part to be delegated. The value is true by default for JDBC and ODBC sources. - Delegate Natural Outer Join (DS: JDBC, ODBC). This indicates whether the source allows for the Natural Outer Join operator to be delegated. The value is false by default for JDBC and ODBC sources. - Delegate NOT Condition (DS: JDBC, ODBC). This indicates whether the source allows the NOT condition to be delegated. The value is true by default for JDBC and ODBC sources. - Delegate OR Condition (DS: JDBC, ODBC). This indicates whether the source allows for the OR condition to be delegated. The value is true by default for JDBC and ODBC sources. - Delegate ORDER BY (DS: JDBC, ODBC). This indicates whether the source allows the ORDER BY clause to be delegated. The value is true by default for JDBC and ODBC sources. - Delegate Projection (DS: JDBC, ODBC). This indicates whether the source allows for projections to be delegated. The value is true by default for JDBC and ODBC sources. - Delegate Register Literal (DS: JDBC, ODBC). This indicates whether the source allows for register-type compound constants to be delegated. The value is false by default for JDBC and ODBC sources. - Delegate Right Function (DS: JDBC, ODBC). This indicates whether the source allows for conditions with functions on the right part to be delegated. The value is true by default for JDBC and ODBC sources. - Delegate Right Literal (DS: JDBC, ODBC). This indicates whether the source allows for conditions with constants on the right part to be delegated. The value is true by default for JDBC and ODBC sources. - Delegate Selection (DS: JDBC, ODBC). This indicates whether the source allows for conditions to be delegated. The value is true by default for JDBC and ODBC sources. - Delegate UNION (DS: JDBC, ODBC). This indicates whether the source allows for the union operator to be delegated. The value is true by default for JDBC and ODBC sources. Visual Creation of Views 96

108 - Supports Modifier in Aggregate Function (DS: JDBC, ODBC). This indicates whether the source supports DISTINCT/ALL modifiers in aggregate functions. - Supports Branch Outer Join (DS: JDBC, ODBC). This indicates whether the source allows for (left right) outer join. The value is false by default for JDBC and ODBC sources. - Supports Eq Outer Join (DS: JDBC, ODBC). This indicates whether the source allows for the Equality Outer Join operator to be delegated. The value is false by default for JDBC and ODBC sources. - Supports Explicit Cross Join (DS: JDBC, ODBC). This indicates whether the source allows for the Explicit Cross Join operator to be delegated. The value is false by default for JDBC and ODBC sources. - Supports Full Eq Outer Join (DS: JDBC, ODBC). This indicates whether the source allows for the Full Equality Outer Join operator to be delegated. The value is false by default for JDBC and ODBC sources. - Supports Full NotEq Outer Join (DS: JDBC, ODBC). This indicates whether the source allows for the Full Not Equality Outer Join operator to be delegated. The value is false by default for JDBC and ODBC sources. - Supports Fusing in Using AND Natural Join (DS: JDBC, ODBC). This indicates if the source merges the same fields when running a natural join or a join with the USING clause. The value is false by default for JDBC and ODBC sources. - Supports Join On Condition (DS: JDBC, ODBC). This indicates whether the source allows for the Join On clause to be delegated. The value is false by default for JDBC and ODBC sources. - Supports Natural Join (DS: JDBC, ODBC). This indicates whether the source allows for the Natural Join clause to be delegated. The value is false by default for JDBC and ODBC sources. - Supports Using Join (DS: JDBC, ODBC). This indicates whether the source allows for the Using Join clause to be delegated. The value is false by default for JDBC and ODBC sources. - Delegate Aggregate Functions List (DS: JDBC, ODBC). This indicates the aggregation functions that can be delegated. In JDBC and ODBC sources, the list is made up of the AVG, COUNT, MAX, MIN and SUM functions. - Delegate Scalar Functions List (DS: JDBC, ODBC). This indicates the scalar functions that can be delegated. - Delegate Operators List (DS: JDBC, ODBC). This indicates the operators that can be delegated. In JDBC and ODBC sources, the list is made up of the =, <>, <=, >, >=, in, between, contains, containsor, like, isnull, isnotnull, istrue and isfalse operators. 5.6 ADVANCED CONFIGURATION OF VIEWS The Advanced button on the schema screen of a view provides access to its advanced configuration wizard (see Figure 72). From this screen, you can configure the view cache, its query capabilities, its swap to disk policy, certain characteristics of its execution plan (only in derived views), its internationalization configuration (only in base views) and some specific aspects for views from some types of sources. The following subsections deal with each of these aspects. Visual Creation of Views 97

109 Figure 72 Advanced Configuration screen of the view average_monthly_sales Internationalization Configuration The internationalization configuration of a base view specifies aspects such as time zones, languages or currencies. For example, the currency to be used in the values of the type Money is specified by this property. DataPort includes the most usual internationalization configurations, but also allows new configurations to be created ad hoc. See Creating New Internationalization Configurations in the Advanced VQL Guide [VQL] for more information Configuring the Cache of a View The following parameters may be used to configure the view cache (caching must have been previously enabled in the server (see section 8.1.3): Time to live. Indicates the time to live (in seconds) for the data stored in the view cache. If the Default checkbox is checked, the default value defined in the server configuration panel will be used (see section 8.1.3). Cache on/off/post. The cache can be activated or deactivated for the view using the cache on/off/post selector. If the option post is selected, the system will detect whether a given query can be answered using another previous query, even where this is not exactly the same as the new query. For example, if the results of a previous query select * from view where field1 = a are in the cache and the system receives the query select * from view where field1 = a and field2 = b, it can be responded to on the basis of the results of the first query, applying a post-processing operation that returns only those tuples where field2 = b. If the on option is selected, the system only uses the cache if the query received is the same as the preceding query. Visual Creation of Views 98

110 Use of the post-processing technique may not be desirable if a wrapper does not always return all results for a query made to a specific source. For example, if a wrapper that accesses a Web source returns only the first 100 results returned by the source for the query select * from view where field1 = a, then the result of applying the post-processing condition (field2 = b) to the results of the query may be different to the result obtained when executing directly on the source select * from view where field1 = a and field2 = b. It is also possible to invalidate the cache content for a view. There are the following options: Invalidate all. Remove all the cache content for the view. Invalidate partially. Remove only the content associated to queries that verify a set of specified conditions. These conditions are created with the conditions editor. Invalidate on cascade. Also, remove the cache of the lower-level views used to create the edited view. This option is supported for both total and partial invalidations. See section 4.5 for information about deleting the cache content of more than one view at the same time. Figure 73 Cache invalidation of a view Swapping Configuration To avoid memory overflows, Virtual DataPort can swap the intermediate results generated during the execution of a query. It can also swap the intermediate results of the sorting operations (views with ORDER BY). Visual Creation of Views 99

111 Before enabling swapping in a specific view, we have to enable the swapping engine in the server (see section for information on how to do it) When a query is being executed, Virtual DataPort will swap to disk when an intermediate result produced during the execution exceeds the maximum size specified. This size can be specified (in megabytes) using the Max size field. Generally, the specified value should not be greater than one third of the memory available for the JAVA virtual machine on which the DataPort server is run. If the Default checkbox is checked, the default value defined in the server configuration panel will be used (see section 8.1.6). NOTE: DataPort can also configure the use of swapping dynamically for a specific query using the VQL CONTEXT clause (see Context clause in the Advanced VQL Guide [VQL]) Editing the Execution Plan A key aspect of query and view run optimization in DataPort is the execution strategy used for each join operation involved. As described in section 5.4.2, it is possible to specify the required execution strategy on creating a join-type view. An execution strategy for a join operation indicates two parameters (see Query Optimization in the Advanced VQL Guide [VQL] for more details): Run method (merge, hash, nested or nested parallel) and Order in which the input relations must be considered. The view at the top will be considered as a first relation, if the Ordered option is chosen, and as second, if Reverse Order is chosen). In addition, it is also possible to choose the ANY option, so DataPort will choose the strategy to apply. However, a join view can be used in defining new derived views and the optimal execution strategy may be different in each case. Example: Suppose there is a join view V3 with attributes A, B and C. V3 has views V1 (with attributes A and B) and V2 (with attributes B and C) as input relations. Let us also suppose that queries on V3 are using the nested execution method. Common optimization practice on using this execution method involves ordering the input relations so that the first is the one to provide least tuples to the join operation. In our example, suppose that the number of input tuples to the join from V1 is normally less than that of V2 and, therefore, V3 is usually queried by considering V1 as the first relation. Now suppose that you want to create a new view V4 that is a selection view on V3 by applying the condition C=c. Also suppose that there are few tuples in V2 that verify this condition. Then, on running a query on V4, the most common situation will be for the join view V3 to be given less tuples from V2 than from V1. Therefore, when V4 is run, V2 should be considered the first relation, unlike the general case where the opposite is sought. To deal with these cases, in any derived view V it is possible to define the preferred run strategy for each join view involved in its definition. These strategies will only be used to run queries on V and will not overwrite the default execution preferences of these join views. Visual Creation of Views 100

112 Figure 74 Editing the Execution Plan of a view Hence, the preferences for the execution plan of V will not affect the queries made directly on the original join views or on other derived views using them. To edit the preferences on the join execution strategy for a derived view, access the advanced configuration screen and click on the Edit Execution Plan button. The execution plan edit screen will then be displayed for the view (the execution plan edit screen is shown in Figure 74 for view inc_grouped_by_pref_clients created in section 5.4.4). The screen displays the view definition tree. For each join operation involved in the definition, it is possible to mark the Specify execution plan (do not inherit it) box to enable two selectable fields to choose the execution method and the order of the input relations required. If the ANY option is chosen for a parameter, DataPort will use internal cost information to choose the most suitable option. On clicking ok, you will return to the advanced configuration screen for the view. By clicking ok again, the configuration changes made on this screen will become effective. NOTE: DataPort can also configure the execution plan preferences for a specific query using the VQL CONTEXT clause (see Context clause in the Advanced VQL Guide [VQL]) Query Capabilities Some data sources such as Web sites or systems that offer a Web Service interface do not allow any query to their data, instead they present interfaces with limited query capability (e.g. HTML forms in the case of a Web site and a list of parameters in the case of a Web Service operation). For this reason, Virtual DataPort allows the administrator to explicitly specify the query capabilities of the base relations. Also, Virtual DataPort automatically calculates the query capabilities of the derived views according to the capabilities of its base views. Visual Creation of Views 101

113 Therefore, when the advanced configuration of a view is accessed, the available options vary according to whether the view is a base relation (also called base views) or a derived view. The difference lies in that in the case of base relations the query capabilities can be modified, while in the case of derived views they can be seen but not directly modified. The query capabilities are expressed using a list of search methods. Each search method specifies a different way to query the view (or base relation). See the next section for a detailed description of how to interpret search methods of a view (or base relation). When a base relation is graphically created, the system automatically assigns search methods to it. While in the majority of cases the methods automatically created do not need to be modified, this may not always be the case, particularly in sources of the type Web and Web Service. Once in the advanced configuration view of a base relation, the default search methods generated for same may be changed easily (see Figure 72). As can be seen in the figure, a search method specifies for each attribute of the view (see the subsection below for more details): Its data type, A list of operators for which the field can be queried. any indicates any operator. New operators may be added for the attribute using the drop-down menu in its Operators column. Operators may also be eliminated using the buttons [x] that appear beside each operator. Whether it is obligatory ( OBL ), optional ( OPT ) or not supported ( NOS ) in the queries to the view. The multiplicity supported for the attribute in queries to the view. A list of the possible values for the attribute that may appear in a condition of a valid query to the view. If no value is specified, it is assumed that every value of the attribute data type is valid. To add new values simply enter them into the text box that appears in the Values column of the attribute and press the button that appears on the right. A value may be deleted using the button [X] that appears at its side. Whether the attribute appears in the response of the queries made to the view. The VQL language may be used directly to add or delete complete search methods (see the Advanced VQL Guide [VQL]) Search Methods Each base relation explicitly describes its query capabilities through what are known as search methods. If a relation has no search method, then no query can be made to it. Each search method is comprised of a series of 5-tuples. Each 5-tuple represents a restriction that a specific query should comply with to be executed on the source using this search method. In this sense, a search method that has no 5-tuple allows every query (this is the case of sources that have no limitations in their query capabilities such as, for example, conventional relational databases). The format of a 5-tuple is (attribute, operators, obligatoriness, multiplicity, possible_values) where: attribute is an attribute of the relation. operators is the group of operators that can be used in conditions with this attribute in the queries to this view and with this search method. ANY represents any operator allowed by the attribute data type. Visual Creation of Views 102

114 obligatoriness can have three values: OBL indicates that the attribute should mandatorily appear in any query to the source. OPC indicates that the attribute may or may not appear in the query (it is optional) and NOS indicates that the queries for this attribute are not allowed in the source. multiplicity indicates how many values can be included in the source query for the given attribute and operator. If it is not possible to make queries for this attribute ( NOS value in the obligatoriness field), the value is necessarily 0. ANY indicates that a query, for any given attribute and operator, could have any number of values. possible_values is the list of values that can be used to query the attribute. If the value ANY is contained in it, this means that the search range is not limited (within the range associated with the attribute data type), and the attribute can be queried about any value. If the obligatoriness field is set in the 5-tuple to the NOS value, then it necessarily takes the value of an empty set. Example: Consider the example of a virtual bookshop on the Internet offering the search form of shown in Figure 75. Figure 75 Search form for a bookshop The form obliges the user to specify a value for the TITLE attribute and gives the option to set a value for the AUTHOR attribute and for the FORMAT attribute (restricted to a group of values). Searches by title and author are searches by keyword (operator like). A search by exact phrase (operator =) is indicated by selecting the box next to the search box of the field. For each attribute, a search can specify one value only. In addition to the fields TITLE, AUTHOR and FORMAT, the shop returns as output a PRICE attribute, which cannot be queried using the form. Let us model this source as a relation R={TITLE,AUTHOR,FORMAT,PRICE} with a search method containing the 5-tuples shown in Figure 76. (TITLE,{like,=}, OBL, 1, Any) (AUTHOR, {like,=}, OPT, 1, Any) (FORMAT, {=}, OPT, 1, { All formats, Hardcover, ebooks, Paperbacks }) (PRICE, {}, NOS, 0, {}) Figure 76 Search method for a bookshop Note that the first 5-tuple has the value {like, =} in the OPERATORS field and OBL in the OBLIGATORINESS field, this does not mean that it is obligatory to query the TITLE attribute with both operators, but that it is obligatory to query it at least with one of them. In order to have the TITLE attribute appear obligatorily in the query with both operators (this is not possible in the form in the example), this should be done with two different 5-tuples for the TITLE attribute, one for each operator: Visual Creation of Views 103

115 {(TITLE, {ANY}, OBL, 1, ANY) (TITLE, {=}, OBL, 1, ANY)}. Thus, as can be seen, when you want to differentiate the treatment of a specific attribute according to the operator with which it is used, more than one 5-tuple can exist for each attribute. When views or relations of the global schema are created, Virtual DataPort can automatically calculate its search methods from those of the sources and the expression used to define the view. This allows the system to know a priori whether or not a specific query can be responded to. Therefore, for derived views DataPort allows the query capacities of same to be consulted, but they cannot be modified directly View Configuration Properties View Configuration Properties allow indicating specific characteristics of the underlying data sources such as their distributed transaction support capacity or whether insert operations are allowed. Section indicated the configuration properties of the data sources. This section describes the configurable properties in each view, depending on the type of data source they have come from. NOTE: Typically, users do not need to edit this information since DataPort automatically uses suitable configurations for most common data sources. The properties of each base view can be configured by selecting the base view from the left-hand menu, either in the DataSources or in the Combine area, and pressing the Advanced button so that the advanced configuration window of the base view appears. Underneath the search methods (using the scroll bar where necessary) is the Wrapper Source Configuration link that leads to the property configuration screen. They can also be configured from VQL (see the Advanced VQL Guide [VQL] for more information). The configurable properties are as follows: - Allow Insert: This indicates whether the data source underlying the view accepts inserting operations. It is applicable to relational databases (accessible via JDBC and ODBC) and CUSTOM wrappers. The possible values are: o Default: VDP assigns a default value depending on the source type. In the case of relational sources and CUSTOM wrappers, the default value is Yes. o Yes: the data source allows for inserting operations. o No: the data source does not allow for inserting operations. - Allow Delete: This indicates whether the data source underlying the view accepts row deleting operations. It is applicable to relational databases (accessible via JDBC and ODBC) and CUSTOM wrappers. The possible values are: o Default: VDP assigns a default value depending on the source type. In the case of relational sources and CUSTOM wrappers, the default value is Yes. o Yes: the data source allows for deleting operations. o No: the data source does not allow for deleting operations. - Allow Update: This indicates whether the data source underlying the view accepts row updating operations. It is applicable to relational databases (accessible via JDBC and ODBC) and CUSTOM wrappers. The possible values are: o Default: VDP assigns a default value depending on the source type. In the case of relational sources and CUSTOM wrappers, the default value is Yes. o Yes: the data source allows for updating operations. o No: the data source does not allow for updating operations. - Delegate All Operators. This indicates whether the view allows for all operators to be delegated. Applicable to CUSTOM wrappers. The value is true by default. - Delegate AND Condition. This indicates whether the view allows for the AND condition to be delegated. The value is true by default for base views from CUSTOM wrappers. - Delegate Array Literal: This indicates whether the view allows for array-type compound constants to be delegated. Applicable to CUSTOM wrappers. The value is true by default. Visual Creation of Views 104

116 - Delegate Compound Field Projection: This indicates whether the view allows projections on compound fields to be delegated. Applicable to CUSTOM wrappers. The value is true by default. - Delegate Left Function: This indicates whether the view allows for conditions with functions on the left part to be delegated. Applicable to CUSTOM wrappers. The value is true by default. - Delegate Left Literal: This indicates whether the view allows for conditions with constants on the left part to be delegated. Applicable to CUSTOM wrappers. The value is true by default. - Delegate NOT Condition: This indicates whether the view allows for the NOT condition to be delegated. Applicable to CUSTOM wrappers. The value is true by default. - Delegate OR Condition: This indicates whether the view allows for the OR condition to be delegated. Applicable to CUSTOM wrappers. The value is true by default. - Delegate Register Literal: This indicates whether the view allows for register-type compound constants to be delegated. Applicable to CUSTOM wrappers. The value is true by default. - Delegate Right Field: This indicates whether the view allows for conditions with fields in the right part to be delegated. Applicable to CUSTOM wrappers. The value is true by default. - Delegate Right Function: This indicates whether the view allows for conditions with functions on the right part to be delegated. Applicable to CUSTOM wrappers. The value is true by default. - Delegate Right Literal: This indicates whether the source allows for conditions with constants on the right part to be delegated. Applicable to CUSTOM wrappers. The value is true by default. - Supports Distributed Transactions: This indicates whether the underlying data source can take part in an XA [XA] distributed transaction. It is applicable to relational databases (accessible via JDBC and ODBC) and CUSTOM wrappers. The possible values are: o Default: VDP assigns a default value depending on the source type. In the case of relational sources and CUSTOM wrappers, the default value is Yes. o Yes: the data source meets the XA specification. o No: the data source does not meet the XA specification. - Data in Order Field List: This indicates the fields by which the data is ordered. They are separated by commas, and the field names are accompanied by the ASC/DESC modifier. Property applicable in all data sources. - Delegate Operators List: This property determines the list of operators that can be delegated to the data source. This allows for VDP to optimize the query plan created using the query made by the user, delegating part of the processing to the native source. While VDP carries out this action automatically on relational databases, other source types do not provide this information in their metadata, despite this sometimes being possible. VDP allows indicating the list of operators that can be delegated in Web Service and CUSTOM data sources (all operators can be delegated by default in CUSTOM wrappers). Operator "=" can be delegated by default in WS wrappers Configuration Properties for Specific View Types The Denodo Aracne (see section 5.5.9) and the Google Enterprise Search / Mini (see section 5.5.8) data sources have specific configuration properties. The specific properties of these types of base views can be configured by selecting the base view from the left-hand menu, either in the DataSources or in the Combine area, and clicking on the Advanced button so that the advanced configuration window of the base view appears. Underneath the search methods (using the scroll bar, if necessary) is the Wrapper Specific Configuration link that leads to the property configuration screen. In the case of base views from Google-type sources, the configurable properties are: Site Collections, Client, Languages and Number of Key match. See section for more information about these properties. In the case of base views from Denodo Aracne-type sources, the configuration properties are those relating to obtaining the most relevant terms of a document (or a field of it). See section for more information about these properties. Visual Creation of Views 105

117 5.7 STORED PROCEDURES Virtual DataPort allows writing stored procedures using the JAVA language (see Advanced VQL Guide [VQL]). The DENODO_HOME/samples/vdp/storedProcedures path includes different example stored procedures. The README file in this path describes their function and contains the instructions to compile and install them. The stored procedure CalculateAvgRevenue, included in these examples, will be used in this section. Note: to follow this example you must: 1. Execute the script $DENODO_HOME/samples/vdp/storedProcedures/scripts/compile_storedprocedur es.bat 2. Load the following Jar file using the Jar Management option: DENODO_HOME/samples/vdp/storedProcedures/target/jars/denodo-demostoredprocedures.jar This option is located in the File Extensions Jar Management menu. See section 8.3 for more information Importing Stored Procedures To import a new stored procedure, open the Combine Projects ProjectName Stored Procedures tab and click the option New. The new stored procedure dialog will be displayed (see Figure 77). The following data are requested on this screen: Name. Name of the new stored procedure. In our example, CalculateAvgRevenue. Class name. Full name of the class that implements the stored procedure. In our example, com.denodo.vdp.demo.storedprocedure.calculateavgrevenue. This class can be obtained from: o o A Jar file in server s file system (click Browse to select it) A Jar loaded into the server with the Jar management option (see section 8.3) In this case, select the Select Jars checkbox. Jar file. Optional. Path to the.jar file containing the class implementing the stored procedure. If the class implementing the stored procedure has not been added to the DataPort server, this option can be used to provide the path to the.jar file that contains it. Select Jars. If selected, select the Jar imported with the Jar Management option that contains the class indicated in the Class name text box. Note: to select two or more Jars, hold the key Ctrl and click on the required Jars. This may be necessary if the stored procedure rely on external jars that have also been imported into DataPort. Visual Creation of Views 106

118 Figure 77 Importing a Stored Procedure On pressing Ok, the base view is created and the Tool displays a dialog with the properties of the new stored procedure and its input and output parameters (see Figure 78). The parameters of a stored procedure can be of types IN, OUT or IN_OUT. The IN parameters are the required input parameters to execute the procedure; the OUT parameters are the ones returned by the execution of the procedure; and the IN_OUT parameters are input parameters that are also returned by the procedure. The Nullable column indicates that the value of that parameter can be NULL. Figure 78 shows information about the new stored procedure: It has an input parameter (IN) called taxid_list, of type array, and an output parameter avgrevenue (OUT), of type double. Visual Creation of Views 107

119 Figure 78 Properties of a Stored Procedure Executing Stored Procedures To execute a stored procedure, press the Execute button from the screen displaying the stored procedure properties. The tool will display a dialog similar to Figure 79 where the user has to fill in the input values; after clicking Ok, the procedure will be executed. Any query execution can be cancelled at any moment. If results are not being shown yet, cancellation can be achieved by pressing the [X] button of the Please Wait dialog which can be seen until the first batch of results is shown. Once the query results are being shown, the execution can be stopped by pressing the Stop button. Figure 79 shows the screen in which the values of the input parameters for the procedure CalculateAvgRevenue are specified. This sample procedure receives as input a parameter of type array of registers called taxid_list. Each register has a field called taxid (client s Tax ID). In our example, we create two registers with the values B and B Visual Creation of Views 108

120 Figure 80 is the result of the execution. Figure 79 Execution of the Stored Procedure CalculateAvgRevenue Visual Creation of Views 109

121 Figure 80 Result of the execution of the stored procedure CalculateAvgRevenue Stored procedures can be also invoked from the VQL Shell using the CALL statement (see Use of Stored Procedures in Advanced VQL Guide [VQL]) Use of Stored Procedures in Creating Views Virtual DataPort stored procedures can be used to create new derived views. More specifically, an invocation of a stored procedure may appear in the FROM clause of any query or view definition. The stored procedure will be considered a view which schema includes all input and/or output parameters (that is, of type IN, OUT, IN_OUT) of the stored procedure, and with a single search method (see section 5.6.5). The search method of a view generated from a stored procedure will have one obligatoriness (OBL) constraint for every input parameter (IN or IN_OUT). To create a view that uses a stored procedure in the FROM clause, drag&drop the procedure on the creation dialog of the new view (union, join, projection/selection, etc) Then, if the procedure includes input parameters, they can be added to the new view by unselecting the Use Input Parameters checkbox (see Figure 81). Otherwise, the user has to fill in the input values. Visual Creation of Views 110

122 Figure 81 Parameters of a procedure added to a new view Visual Creation of Views 111

123 6 PUBLICATION OF WEB SERVICES 6.1 WEB SERVICE TYPES Virtual DataPort can publish the content of one or several views (and/or stored procedures) as a Web service to provide access to the information of these views from any external application. It can also publish the result of any VQL statement. These Web services can be deployed in the Web services container embedded in the Denodo Platform or deployed in an external application server. The Web services can be published in the following versions: SOAP [SOAP] Web Services. REST-style Web Services that use HTTP directly as the transport protocol and return data encoded in XML. JSON Web Services. Similar to the REST-style Web services, although the output will be in JSON format [JSON]. RSS Web Services. Similar to the REST-style Web services, although the output will be produced in the RSS format. HTML Web Services. Similar to the REST-style Web services, but the output is an HTML table containing the response data for the query executed. The table includes JavaScript code to sort the results by any field and/or paginate the returned results. It is also possible to adjust the size of the table and the cells and to modify its graphic appearance using a CSS file. See section 7 for more options to generate visual widgets from Virtual DataPort views and/or stores procedures. The following section describes the publication process for these Web Services. Figure 82 Publishing a Web Service Publication of Web Services 112

124 6.2 PUBLISHING WEB SERVICES Views / Stored procedures / VQL statements are published using the Publish Web Services tab in the Administration tool. This tab displays the status of the Web Services container embedded in the Denodo Platform and the list of Web services currently created (see Figure 83). A service can be deleted by clicking on the Drop option alongside its name in the drop-down menu on the left of the screen. The following information is provided for each service: Name of the Web service. Status. This indicates whether the service is deployed or not. If it is deployed but changes have been made to the configuration of the service after deploying it, this is indicated by an *. Context path. Root path to the service in the embedded application server. Export. Contains links to generate the different artifacts associated with the Web service (.war file and/or wsdl file). Both files generated will be accessible for downloading at the /export path of the Web container embedded in the Platform (e.g. o The.war file contains the implementation of the Web Service in the versions chosen during publishing. This is useful if you want to deploy the service in an external JEE Web container (such as Apache Tomcat [TOM]). Before exporting, you will be asked for the URI of the DataPort server and the user name and password to access it. o The.wsdl file [WSDL] defines the interface of the SOAP version of the Web Service generated. It can be used with a utility for SOAP Web Services programming (such as those included in Apache Axis [AXIS]) to generate the necessary stubs to implement a client program accessing the SOAP Web service. For information on how to access the REST, JSON, RSS, and HTML versions of the Web Service see section 6.3. Deployment. If the service is not deployed in the Web container embedded in Virtual DataPort, the available option is Deploy. If it is already deployed, the options are Redeploy or Undeploy. Publication of Web Services 113

125 Figure 83 Web services The steps to generate a new Web Service are: 1. Click on the new option of the Web Services tab (see Figure 82). 2. Select a view (or stored procedure) to export using the Projects drop-down menu on the left and drag&drop to the work area on the right of the screen. In our example, drag&drop the inc_grouped_by_pref_clients view created in section Alternatively, you can also click on the Publish from vql expression button to add an operation that will return the result of a VQL statement. This statement can include interpolation variables that will be considered the input parameters of the Web Service operation. See Execution context of a query and interpolation strings in the Advanced VQL Guide [VQL] for more information about interpolation variables. 4. When adding the view or stored procedure, DataPort analyzes the query capabilities of it and generates a series of candidate operations to query and/or update the view through the Web Service (Figure 84 shows the operations generated for our example). The generated operations have these characteristics: The input parameters of the generated operations correspond to the attributes of the view or the input parameters of the stored procedure. In our example, the generated operation is getinc_grouped_by_pref_cl, which receives three input parameters that correspond to the attributes of the view: taxid, revenue and num_incidences. The return value of the operations generated will be an array of elements. Each of these elements will correspond to a tuple of the view and will have one subelement for each attribute. Publication of Web Services 114

126 If the view published is updateable, default operations will also be generated to insert, update and delete tuples from the view (see section for further details). Figure 84 Operations created for the view inc_grouped_by_pref_clients 5. It is possible to modify operations as follows: Delete an operation: click on the Drop option alongside its name. Change the operation s name: clicking on the Rename option. Delete an input parameter of an operation: click on the Drop option alongside the name of the parameter). Depending on the query capabilities of the view (see section 5.6.5), some operations generated may include mandatory parameters that cannot be deleted. Delete a return parameter. Indicate the operator used to query DataPort with the value provided for this parameter: click on the current operator and select the new one. 6. A Web Service can include operations based on more than one view. To do so, simply drag & drop from the new view and repeat steps 2 to Change the name of the new Web Service: fill in the Edit Web Service Name text box and click on the arrow button. 8. Click on the Advanced button to configure additional settings of the Web Service publication: Access types to be generated for the service: SOAP, REST, JSON, RSS, and/or HTML. For SOAP Web services only, the Web Service style: RPC or DOCUMENT. Some Web Service consumer applications may require a specific style. Publication of Web Services 115

127 For SOAP, REST and/or JSON Web services, the type of data to be used for exporting the datetype fields. The available options are the XML Schema date and datetime data types. The XML Schema date type only considers the day, month and year, whereas the datetime includes full information on time, including hours, minutes, seconds, and milliseconds. SOAP XSLT Transformations. We can transform the SOAP requests and responses to adapt them to the requirements of an existing SOAP client. See section REST XSLT Transformations. We can transform the REST responses to adapt them to the requirements of an existing client. See section SOAP over JMS. Virtual DataPort can subscribe to a JMS server to listen to SOAP messages. See section CSS File. For Web services with HTML output, it is possible to specify a CSS file that adapts the graphic appearance of the HTML output generated by DataPort. By accessing this option, it is possible to view the CSS document to be used and load a new one, if required. RSS Mappings. The RSS format imposes a series of specific fields for each item of its output. Therefore, on exporting a view in RSS format, the correspondence between the fields of the view and the fields in RSS format must be specified. See section for further details. Configuration of the authentication of the new Web Service. See section Configuration parameters for connections. The Web services published will be deployed to an application server (usually in the Web Service container embedded in the Denodo Platform). Each time a Web service operation is invoked, it will produce a statement executed against the DataPort server to obtain the required results. It is possible to configure certain connection parameters to be used by the Web service to access DataPort (see section for further details). 9. Click on Ok to create the Web service. To deploy it, click on Deploy in the table of existing Web Services. After creating a Web service, you can open it by clicking on its name in the Publish tab. Then, you can: Add more views/stored procedures to publish. Edit the existing operations of the Web service Change the Advanced settings of the Web service. View the VQL code that generates the Web service by clicking on View VQL. This dialog has the same options as the View VQL dialog of views (see section ), except Show dependencies. This dialog does not have this option because it only displays the VQL code of the Web service Operations for Updatable Views If the published view is updateable (see Inserts, Updates, and Deletions section in the Advanced VQL Guide [VQL]), the Tool will generate operations to insert, update, and delete tuples from that view. More specifically, the following operations will be generated: Publication of Web Services 116

128 An insert operation with a parameter for each field of the view. The name of the operation will start with insert followed by the name of the view (e.g. if the view is called INTERNET_INC, the operation will be called insertinternet_inc). A delete operation with a parameter for each field of the view. The name of the operation will start with delete followed by the name of the view (e.g. if the view is called INTERNET_INC, the operation will be called deleteinternet_inc). An update operation with two parameters for each field of the view: one of the parameters will take the name of the field and the other will take the name of the field prefixed by New. The parameters with the prefix New correspond to the new values (SET clause of the UPDATE statement). Whereas those without this prefix are used to choose the tuples to be updated (WHERE clause of the UPDATE statement). The name of the operation will start with update followed by the name of the view (e.g. if the view is called INTERNET_INC, the operation will be called updateinternet_inc). IMPORTANT NOTE: On including insert / update /delete operations in the services published, you are enabling external clients to insert, delete or update tuples. If you want to avoid this, use the Drop option alongside each operation to delete them from the Web service published Configuration of Connections in the Web Services Published The Web services exported will be installed in an application server (typically in the Web service container embedded in the Denodo Platform). Each time a Web service operation is invoked, it will execute a statement against the DataPort server to obtain the required results. It is possible to configure certain connection parameters to be used by the Web service to access DataPort. The most important aspect that can be configured is the use of a connection pool. Hence, the connections opened by the Web service with DataPort may be reused. In production environments, the use of the connections pool is strongly recommended. Connection parameters are configured in the Connection Parameters section of the advanced configuration of the publication process of a new Web service. The parameters that can be modified are: Chunk Size, Chunk Timeout, and Query Timeout. Their interpretation is the same as in any other VDP client (see VDP Developer s Guide [DEV]). Enable Pool. This box must be marked to enable use of the connection pool. Initial Size. Initial number of connections to be opened in the pool. Max Active. Maximum number of connections in the pool. A negative value means there is no limit SOAP XSLT Transformations In an environment with existing SOAP clients and services, we do not need to modify those clients to work with Virtual DataPort Web services. We can define XSLT stylesheets [XSLT] to transform the incoming SOAP messages to adapt them to the format that the new Web service expects. We also can define stylesheets to transform the SOAP responses before sending them to the existing clients. Follow these steps to define XSLT transformations for a Web service: 1. Open the Advanced configuration of the Web service. 2. Click on SOAP XSLT Transformations. Publication of Web Services 117

129 3. Select an operation and click on Add Transformation 4. Select the Input XSLT Transformation box and write the stylesheet. This stylesheet will be applied to the incoming SOAP messages to transform them into the format expected by the Virtual DataPort Web service. 5. Select the Output XSLT Transformation box and write the stylesheet. This stylesheet will be applied to the SOAP responses generated by the Web service to transform them into the format expected by the existing SOAP client. 6. If the existing SOAP client sends the parameter SOAPAction with its requests and the parameter cannot be changed, select the Incoming SOAPAction box and write it. 7. Repeat steps 3 to 6 for each operation of the Web service that we want to transform. See appendix 12.7 for more details on how to generate these XSLT stylesheets REST XSLT Transformations In an environment with existing REST clients, we do not need to modify those clients to work with Virtual DataPort Web services. We can define XSLT stylesheets [XSLT] to transform the XML responses of the Virtual DataPort REST services, before sending them to existing clients. Follow these steps to define XSLT transformations for a Web service: 1. Open the Advanced configuration of the Web service. 2. Click on REST XSLT Transformations. 3. Select an operation and click on Add Transformation. 4. Select the Output XSLT Transformation box and write the stylesheet. This stylesheet will be applied to the REST responses (XML) generated by the Web service to transform them into the format expected by the existing client. Repeat steps 3 and 4 for each operation of the Web service that we want to transform SOAP over JMS SOAP is transport-independent and can be bound to any protocol. Although it is usually used with HTTP, it can also be used with JMS (Java Message Service [JMS]). When using SOAP over JMS [SOAP_JMS], the client sends the SOAP message to the JMS server, which forwards it to the Web service. Then, the Web service sends the response back to the JMS server, which forwards it to the client. SOAP over HTTP is more interoperable as there is more support for it. However, there are certain factors that we can only achieve using SOAP over JMS: a) Scalability. By using SOAP over JMS, clients can send requests without waiting for the response. That way, the server does not have to process all the requests as they arrive and clients are not blocked waiting for a response. b) Reliability. The JMS server ensures that requests and responses are delivered. In case of any failure in the communication, the JMS server keeps trying to send the messages. This is important in environments that deal with critical data. Publication of Web Services 118

130 Virtual DataPort can subscribe to a JMS server to listen to SOAP messages. Follow these steps to enable this feature: 1. Open the Advanced configuration of the Web service. 2. Click on SOAP over JMS 3. Select On to enable SOAP over JMS support 4. Destination is the name of the queue or topic that Virtual DataPort will subscribe to, waiting for SOAP messages. Depending on the vendor of the JMS server, we might have to create the destination, or it is created automatically when the new Web service tries to subscribe to it. 5. Select Queue or Topic. 6. User name and Password. Credentials to connect to the JMS server. 7. JMS vendor. Select GENERIC if the vendor of the JMS server is not in the list and it can be accessed via JNDI. In this case, we have to provide the appropriate JNDI connection properties (see appendix 12.8). 8. Click Ok. When we deploy the Web service, it connects to the selected JMS server and subscribes to the JMS queue or topic, so it can receive the messages sent to that destination. IMPORTANT NOTE: a Web service with SOAP over JMS enabled needs some client jars to be able to connect to the JMS server. The vendor provides these jars. Appendix 12.8 contains lists of required client jars for the most popular vendors. The first time a Web service is deployed into the embedded Web container of Virtual DataPort, copy these jars to the directory $DENODO_HOME/resources/apache-tomcat/common/lib. Then, restart the Virtual DataPort server so it can load the Java classes of these jars. If the Web service is deployed into an external application server, we have to copy these jars into the /WEB-INF/lib directory of the war file, before deploying it Mappings in the Publication as RSS The RSS 2.0 format [RSS] specifies a series of specific fields for each item. Therefore, on exporting a view in RSS format, the correspondence between the fields of the view and the fields in RSS format must be specified. The following options are displayed on the RSS mappings screen: If the service published includes operations of more than one view (or stored procedure), the RSS version of the service may only include the operations to query one of them. The selectable at the top left enables you to choose the view or stored procedure to be used. Values for channel attributes. An RSS feed contains a channel element that specifies general information on the feed. In this group of options, it is possible to specify constant values for each of the channel subelements permitted by RSS format. Mappings for the attributes of channel items. An RSS feed contains a list of item elements. DataPort will generate an item element for each row returned by the query executed on the view or stored procedure selected. This group of options enables you to select the attribute of the view that corresponds to each Publication of Web Services 119

131 item subelement defined in the RSS format. If the mapping for a certain item subelement is left with the none value, that subelement will not be included in the output feed. The RSS format specifies that at least one value must be assigned either to the title subelement or to the description subelement Web Services Authentication The available authentication options depend on the service type: For REST, JSON, HTML and RSS the options are: None. The Service can be accessed without authentication. HTTP Basic. Basic HTTP authentication [HTTP-AUTH] with the credentials passed as plain text. All the users will use the values of the fields Login and Password as credentials for accessing this Service. HTTP Basic with VDP. Basic HTTP authentication using the credentials of the Virtual DataPort s users. That is, the Web service will connect to Virtual DataPort with the credentials used by the client of the Web service. Only users whose user name is in the Accepted user(s) list will have access to the Service (separate user names with commas). If this list is empty, the Web Service will accept all Virtual DataPort s users. Unlike with the other authentication methods, with this one, we have to grant the user privileges to access the published views. HTTP Basic with LDAP. Basic HTTP authentication using an LDAP server. To configure this authentication follow these instructions: 1. Select the LDAP data source representing the LDAP server used for authenticating users. See section to learn how to create an LDAP data source. 2. Fill in the User pattern text box. The User pattern is used to build the user's Distinguished Name replacing token with the received user name. I.e. if the User pattern is cn=@login,ou=people,dc=yourorganization,dc=com, token will be replaced by the user name provided by the invoker of the Service. 3. Accepted user(s): only users whose user name is in this list will have access to the Service. To accept more than one user, separate their user names by commas. If empty, every user authenticated by the LDAP Server will be granted access to the Service. HTTP Digest. HTTP Digest access authentication [HTTP-AUTH] The authentication options for SOAP Web Services are the same as for REST, JSON, HTML and RSS, but also support the Web Services Security protocol [WSS] Types Conversion Table for REST / SOAP Published Web Services Table 1 shows the conversions applied between the DataPort data types and the data types used for the input and output parameters of the Web services generated in the SOAP and REST versions. In this table, the xsd prefix identifies the XML Schema types and the soapenc prefix, the SOAP types. DataPort Data Type SOAP Web Service Data Type REST Web Service Data Type text xsd:string xsd:string long xsd:long xsd:long double xsd:double xsd:double Publication of Web Services 120

132 int xsd:int xsd:int boolean xsd:boolean xsd:boolean link xsd:string xsd:string float xsd:float xsd:float money xsd:double xsd:double time xsd:int xsd:int date xsd:date or xsd:datetime xsd:date or xsd:datetime blob xsd:hexbinary xsd:hexbinary xml xsd:string xsd:string enumerated xsd:string xsd:string array register xsd:complextype (restriction on soapenc:array) xsd:complextype (sequence) xsd:complextype (sequence) xsd:complextype (sequence) Table 1 Conversions between DataPort data types and Web service parameter types 6.3 INVOKING THE EXPORTED WEB SERVICES This section describes how to invoke the different types of Web services. The Context Path column of the list of published Web services contains the path to each Service. This path contains a list of the exported versions for that service. The relative paths /services, /rest, /json, /html and /rss show the available operations for each Web service type. Note: In the following examples, we will suppose that the embedded Web Container is running on port 9090 of the acme host and the name of the exported web service is testws of the admin database. Example: the following table contains a list of the URLs of the information pages: WS Version SOAP REST JSON HTML RSS URL Table 2 URLs of Web Service s information pages These information pages list the operations of the Web Service. The SOAP information page contains a link to the wsdl file of the service. The REST information page contains a link to the XML Schema of the XML output of each operation. The links to the XML Schemas of the REST services output have this format: Publication of Web Services 121

133 Example: If the testws Web service has an operation called getinternet_inc, the URL of the XML Schema of its response is: Table 3 Obtaining the XSD Schema of a REST operation The URL to invoke an operation of a REST service has this format: paramname1=value1&...&paramnamen=valuen, where N is the number of parameters of the operation. The format for the HTML, JSON and RSS services is the same but replacing rest by, respectively, html, json and rss. Note: SOAP Web Services should not be used invoking an URL, but using a SOAP client instead. Example: Let us suppose that the testws service has an operation called getinternet_inc that requires no parameters. The operation can be invoked as follows: WS Version REST JSON HTML RSS URL Table 4 Invoking a Web Service without parameters Example: Let us suppose that the testws service has an operation called getinternet_incbyiincid that requires one input parameter called iinc_id. The operation can be invoked as follows: WS Version REST JSON HTML RSS URL getinternet_inc?iinc_id=1 getinternet_inc?iinc_id=1 getinternet_inc?iinc_id=1 getinternet_inc?iinc_id=1 Table 5 Invoking a Web Service with one parameter If some parameter is of compound-type, its value will be represented by using the ROW and {} VQL constructors (see section Conditions with compound values in VQL Advanced Guide [VQL]). Example: Let us suppose that the testws Web service has an operation called getrevenuesum. This operation publishes the view REVENUESUM created in section It has a record array-type input parameter Publication of Web Services 122

134 called clients. Each record has one text-type field that represents a company s tax identifier. The operation returns the sum of the revenue of each company in the input parameter. The URL to invoke the REST version of this operation is: clients={row('b '),row('b ')} Table 6 Invoking a REST Web Service with an array-type parameter By default, the REST, JSON and HTML Web Service return an error if they receive parameters that do not belong to the published view. If the Web Service client needs to ignore extra parameters, it must add the validateparams parameter to the URL. For example, if a client invokes the URL getinternet_inc/?made_up_parameter=1 the Service returns an error because made_up_parameter does not belong to the published view. If the client invokes getinternet_inc/validateparams/false?made_up_parameter=1 the Service will ignore made_up_parameter and return the result of the query. The param validateparams has to be added to the URL after the name of the operation and is valid for the REST, JSON and HTML Web services HTML Output Configuration The HTML version of the Web services can be invoked with additional parameters in order to configure the HTML table that displays the results of the queries: shownumresults. If true, the table will also display the number of rows obtained by the query. intervalsize. If present, the results of the query will be paginated. The value of the parameter is the number of results in each page. maxresults. Maximum number of results to be displayed. If the query returns more rows, all excess results will be omitted. cellwidth. Maximum cell width expressed in number of characters. If the text of a cell is wider, the text will be divided in several lines. cellheight. Maximum number of lines in a cell after having splitted the text according to the cellwidth parameter. If this is exceeded, all the cells of this column will have a scroll bar. width. Maximum width (in pixels) of the table. If the size is exceeded, a scroll bar is added. height. Maximum height (in pixels) of the table. If the size is exceeded, a scroll bar is added. Publication of Web Services 123

135 noescapehtml. List of the names of the columns which HTML code will not be escaped (separate each name with comma). By default, the HTML of all the cells is escaped, unless its column name is in this list. These parameters must be indicated in the part of the URL corresponding to the access path (before the query parameters) in the following format: paramname1/value1/.../paramnamen/valuen For example, the following expression invokes the getinternet_inc operation, limiting the number of results displayed to 50 and setting the number of rows per page to maxresults/50/intervalsize/10/?iinc_id=1/ Table 7 Invoking the HTML Web Service with configuration parameters JSON Output Configuration The JSON version of the Web service can return JSON data, prefixed with the name of a function (also called JSON with padding or JSONP). That way, when a browser receives the response, it receives a script rather than data. To do it, add the parameter jsoncallback to the parameters of the URL. E.g: jsoncallback=js_function Table 8 Invoking the JSON Web Service with padding (JSONP) This will return the following: js_function( <result of the query> ); Note: to use this feature, the published view cannot have a field called jsoncallback. Publication of Web Services 124

136 7 PUBLICATION OF VIEWS AS WIDGETS Virtual DataPort allows a view or stored procedure to be published as a visual widget. Then, these widgets can be exported to different widget technologies: Java Portal Servers (Portlets JSR-168 [JSR-168] and JSR-286 [JSR-286]) Microsoft SharePoint Web Parts [SHAREPOINT] OpenAjax [OPENAJAX] These widgets allow querying and displaying the data of the selected view and can interact with other widgets sending them the data obtained from the view or receiving it to filter the contents of the view that are shown. 7.1 PUBLISH A VIEW AS A WIDGET Widgets are created and published from the Publish Widgets tab of the Administration Tool. When accessing this tab, we can see a table containing the list of all the existing widgets. To create a new widget, click on the New button of the Publish Widgets tab. The following information must be provided: Widget name. Name of the new widget. Display name. Name that will be displayed to the users. Element to publish. Select the view/stored procedure to publish by selecting it in this drop-down list. Alternatively, it is possible to: o Drag & Drop or click on the element in the Projects list in the left panel. o Use the Quick Search. Target project. Project where the new widget will be stored. Configure the Advanced settings of the widget: o o o Enable Help Mode. The three widget technologies have a Help Mode that, if enabled, displays information about how to use the widget. There is a default help text that can be changed by clicking on Edit help mode content. Custom Table Events (only for JSR-286 portlets). Select this option to enable the Full Table JSR-286 complex event. Note: increase the performance of the JSR-286 portlets by deselecting this option (see 7.3 for more detail). Connection parameters. Configure the parameters used by the portlets and the auxiliary Web Service to connect to Virtual DataPort (see section 0 to determine the auxiliary Web Service of the widget) It is usually correct to use the default values for these parameters. Chunk Size. Maximum number of results that a block can contain. If this parameter is 0 or empty, all the results will be returned in a single block. Chunk Timeout. Maximum time (in milliseconds) the server will wait before returning a new block. After that time, the current block will be returned even if it does not contain the number of results specified by the Chunk Size parameter. Publication of Views as Widgets 125

137 If this parameter is 0 or empty, all the results will be returned in a single block whenever the query execution ends. Query Timeout. Maximum time (in milliseconds) the server will wait for the termination of a query. If its value is 0, it will wait indefinitely until the query finishes. Enable Pool. Enable the use of the connection pool. In production environments, the use of the connections pool is strongly recommended. Initial Size. Initial number of connections to be opened in the connection pool. Max Active. Maximum number of connections in the pool. A negative value means there is no limit. Click Ok and click on the Publish Widgets tab to open the Widgets Status Table (see Figure 85) Figure 85 Widgets Status Table This table provides the following information for each widget: Name of the widget. Export. Contains links to export the widget to different widget technologies. Status indicates whether the auxiliary Web Service is deployed in the embedded Web container. If it is deployed, but changes have been made to the widget configuration, this is indicated by an *. Context path. If the auxiliary Web Service is deployed in the embedded Web container, contains the path to the service. Deployment. Contains links to deploy / redeploy / undeploy the auxiliary Web Service of the widget. Properties. Click on Show to display the properties of the widget. Publication of Views as Widgets 126

138 7.2 AUXILIARY WEB SERVICES The Open Ajax Widgets and the Microsoft Web Parts cannot directly connect to the Virtual DataPort server, as the Java Portlets do. Therefore, they require an auxiliary Web Service that will retrieve the data from DataPort and send it back to these widgets. Every time a widget is created, its auxiliary Web Service is also created, but they still need to be deployed. There are two options to deploy an auxiliary Web Service: 1. Deploy it in the embedded container: click Deploy in the Deployment column of the Widgets Status panel. 2. Export it to an external J2EE Web Container (such as Apache Tomcat [TOM]): click war in the Deployment column of the Widgets Status panel and follow the required steps to deploy a web application in your specific environment. Note: It is required to deploy the auxiliary Web Service before using an OpenAjax Widget or a Microsoft Web Part. Otherwise, these widgets will not display any data. 7.3 EXPORT TO JSR-168 OR JSR-286 PORTLET A widget can be exported to a JSR-168 or JSR-286 portlet, so it can then be deployed it in any standard Java Portal Server. When clicking the JSR-168 or the JSR-286 link in the Export column of the Widgets Status Table, the user will have to fill in: URI of the DataPort server that the portlet will obtain the data from. Credentials to access to it. This will generate a.war file ready to be deployed in any standard Java Portal Server. This file can be obtained from: $DENODO_HOME/resources/apache-tomcat/webapps/export/<name of the widget>.war Or, of the widget>.war This portlet will display the contents of the published view or stored procedure. In the EDIT mode, the portlet displays a query form where you can configure (see Figure 87): Query parameters that will be used to filter the contents of the exported view. Hide some of the columns of the results table. Results table settings: rows per page, height and width of the results table and its cells, and columns in which HTML code will not be escaped (if a cell of a selected column contains HTML, it will not be escaped) The main difference between the two types of portlets is that the JSR-168 version can only display the content of a view. On the other hand, the JSR-286 version can interact with other portlets by sending/receiving events or sharing public render parameters (see [JSR-286] to learn more about these terms) The JSR-286 portlet can interact with other portlets by: - Sending simple events. When the user selects a row of the table by clicking the icon of the first cell of the row (see Figure 86), the portlet sends a simple event for every cell of that row. The payload of each event is the content of a cell. Publication of Views as Widgets 127

139 - Sending row events. When the user selects a row of the table, the portlet sends an event containing an object of the type com.denodo.vdb.wsgenerator.common.execution.portlet.vo.datarow This object contains the data of the selected row. - Sending table events. When the user clicks on the icon in the first cell of the header, the portlet sends an event containing an object of the type com.denodo.vdb.wsgenerator.common.execution.portlet.vo.datatable This object contains all the data of the results table. To be able to develop a portlet capable of receiving the DataRow and the DataTable events, you need to add a reference to the library $DENODO_HOME/lib/portlet-client-core/portlet-client-core.jar - Receive an event. The JSR-286 portlet can receive an event for every parameter of the query form. The name of these events follows the convention Input_ <name of the search parameter> - Sharing public render parameters. There is one render parameter for each column of the results table and a render parameter for each parameter of the search form. Publication of Views as Widgets 128

140 Click to send an event with all the data of the row and 3 events containing the value of each cell (DataRow) Click to send an event with all the data of the table (DataTable) Figure 86 JSR286 Portlet Publication of Views as Widgets 129

141 Figure 87 Portlet search form 7.4 EXPORT TO OPENAJAX WIDGET A Widget can be exported as an OpenAjax widget and then deployed in an OpenAjax compliant container such as the OpenAjax Mashup Editor [OPENAJAX]. When clicking the OpenAjax link in the Export column of the Widgets Status Table, the user will have to fill in: URL of the auxiliary Web Service of this widget. The default value of this field is the path of the auxiliary Web container, embedded in the DataPort server. So, if the auxiliary Web Service will be deployed to the embedded container, the default value is correct. Deployment URL: URL of the directory where the OpenAjax Widget will be located in the server where the OpenAjax container is deployed. This will generate a.zip file ready to be deployed in the OpenAjax container. This file can be obtained from: Publication of Views as Widgets 130

142 $DENODO_HOME/resources/apache-tomcat/webapps/export/<name of the widget>.zip Or, of the widget>.zip Note: Remember to deploy the auxiliary Web Service. 7.5 DEPLOYMENT OF AN OPENAJAX WIDGET Before deploying an OpenAjax widget, you must install an OpenAjax container (see appendix 12.5 for instructions on how to install the OpenAjax Mashup Editor). In this section, it is explained how to deploy the widget in the OpenAjax Mashup Editor (consult your container documentation if you are using another OpenAjax container): Follow these steps to deploy a new OpenAjax widget: 1. Copy the.zip file generated in the previous section to the $MASHUP_EDITOR_HOME/gadgets/samples/gadgets 2. Unzip the file into a new directory with the same name as the widget. I.e.: if the widget is called internet_inc, the name of the new directory has to be internet_inc. 3. Open the Repository Browser: 4. Click on the Add button at the bottom of the screen and a dialog like the one in Figure 88 will be displayed. 5. Click Browse and select the OpenAjax metadata file located in $MASHUP_EDITOR_HOME/gadgets/samples/gadgets/<name of the gadget>/<name of the gadget>_oam.xml 6. Fill in the Version text box (the Version number is not important) 7. Click Save. Publication of Views as Widgets 131

143 Figure 88 Adding a new item to the OpenAjax Repository To open the OpenAjax Mashup Editor, follow these steps: 1. Open the URL 2. Click on the down-arrow on the left upper side of the Mashup Tool, browse through the menu to locate the new widget and click on its name (see Figure 89) Publication of Views as Widgets 132

144 Figure 89 Adding the new widget to the OpenAjax Mashup Editor Now, the new widget is added to the Mashup (see Figure 90) Publication of Views as Widgets 133

145 Figure 90 OpenAjax Widget displaying the content of the incidents_sales view. To open the widget s search form, click on the down arrow on the upper left side of the widget and click on Edit widget preferences (see Figure 91) In the search form (see Figure 92) you can configure: Query parameters that will be used to filter the contents of the exported view. Hide some of the columns of the results table. Results table settings: rows per page, height and width of the results table and its cells, and columns which HTML code will not be escaped (if a cell of a selected column contains HTML, it will not be escaped) Once you have finished, click Search to go back to the results table. Publication of Views as Widgets 134

146 Click on this icon to open the widget s menu Figure 91 Opening the OpenAjax widget configuration form Publication of Views as Widgets 135

147 Figure 92 OpenAjax widget search form To interact with other widgets you need to open the Edit Widget Properties dialog (see Figure 93). To do this, click on the down arrow on the upper left side of the widget and click on Edit Widget Properties. In this dialog, you can bind the properties of the search form (properties that start with Input_ ) with properties of other widgets. This is useful if you want to filter the contents of the published view by the values received from another widget. You can also bind the value of the cells of the selected row with another widget. In Figure 94, the user is binding the property SUMMARY with the property message of another widget. After doing this, when the user selects a row (clicks on the icon of the first column of a row), the value of the cell SUMMARY for that row will be sent to the other widget. Publication of Views as Widgets 136

148 Click to bind this property with another widget s property. The received value will be used to filter the contents of the results table. This property contains the value of the parameter TAXID of the search form. This property contains the value of the column SUMMARY of the selected row. Click to bind this property with the property of another widget. Figure 93 Edit Widget Properties dialog: binding properties of a widget with properties of other widgets Publication of Views as Widgets 137

149 Figure 94 Binding the property of a widget with the property of another widget. 7.6 EXPORT AS MICROSOFT WEB PART A widget can be exported as a Microsoft Web Part and then deployed in a Microsoft Office SharePoint Server [SHAREPOINT]. When clicking the Web Part link in the Export column of the Widgets Status Table, you will have to fill in the URL of the auxiliary Web Service of this widget. The default value of this field is the path of the auxiliary Web container, embedded in the DataPort server. So, if the auxiliary Web Service will be deployed to the embedded container, the default value is correct. After clicking Ok, DataPort will generate a.zip file containing a.webpart file and an.xml file that contain the necessary information for displaying the view. The new file can be obtained from: $DENODO_HOME/resources/apache-tomcat/webapps/export/<name of the widget>.zip Or, of the widget>.zip Publication of Views as Widgets 138

150 7.7 DEPLOYMENT OF A MICROSOFT WEB PART The first time you want to deploy a Web Part you have to deploy the Denodo SharePoint Solution located in $DENODO_HOME/webapps/webpart-core/DenodoWebPart wsp. See appendix After this, we are ready to deploy the exported Web Parts. To do this, follow these steps: 1. Unzip the exported.zip file in your SharePoint server and copy the.xml file to: C:\Inetpub\wwwroot\wss\VirtualDirectories\80\wpresources\DenodoW ebpart 2. Open the Web Part gallery, usually located in and load the.webpart file into the gallery (see Figure 95) Now, the Web Part is ready to be added to any page Figure 95 Loading the.webpart file into the Web Part gallery. As well as portlets and OpenAjax widgets, the Web Parts also have a query form to configure the following parameters (see Figure 96): Query parameters that will be used to filter the contents of the exported view. Hide some of the columns of the results table. Results table settings: rows per page, height and width of the results table and its cells, and columns which HTML code will not be escaped (if a cell of a selected column contains HTML, it will not be escaped) Publication of Views as Widgets 139

151 Figure 96 Web Part search form Publication of Views as Widgets 140

152 8 SERVER ADMINISTRATION This section describes the main options available to administer the Denodo Virtual DataPort server. Firstly, details are given about how to configure various aspects of the Virtual DataPort server. Then, how to export the Server metadata for backup and migration purposes is described. 8.1 CONFIGURING THE SERVER To manage the databases, users and access rights of a Virtual DataPort server it is necessary to access the administration tool in server administration mode, which implies doing it with an administrator-type user. When accessing the server for the first time, the default administrator-type user can be used (user login admin with password admin ). The server configuration parameters are explained in the following sections: Server connectivity: section Pool of threads: section Cache: section Limitation of concurrent requests (query queueing): section HTTP proxy: section Swapping parameters: section Execution parameters of stored procedures: section Internationalization configuration: section JMS listeners: section Server Connectivity Open the Server Configuration Server Ports dialog (see Figure 97) to configure several connectivity settings of the Virtual DataPort server: Port numbers of the Virtual DataPort server: execution, shutdown and auxiliary ports. The Server Port Number is the port where the server listens to connections from the Administration Tool, JDBC clients and clients using the Virtual DataPort API. The ODBC Port Number is the port where the server listens to ODBC connections. Port numbers of the Web container embedded in the Denodo Platform. Important note: when the Virtual DataPort Server and the Administration Tool or other clients are installed in different machines, you may need to change the interface that the Server listens to connections. To change this, open the Denodo Control Center, open the Configuration dialog and change the RMI Host of the JVM Options. See more details in the section Virtual Machine and Web Container configuration of the Installation Guide [DENINST]. The reason for this is that in some environments, when listening to localhost, the server does not allow connections from remote hosts. Server Administration 141

153 Note: When the connection between clients and the Virtual DataPort server is established through a firewall, this must be configured to allow the connections to the execution and auxiliary ports. Figure 97 Port Configuration The ports changes will take effect the next time the Virtual DataPort server is launched Threads Pool As mentioned earlier, whenever possible, the Virtual DataPort query execution engine operates concurrently on the data sources. Implementation of the concurrence is carried out through a multithreading subsystem that can be configured through the Virtual DataPort administration tool. To configure the multithreading subsystem of a Virtual DataPort server, access Server Configuration - Threads Pool. The configurable parameters are as follows: Max Threads: Maximum number of threads the pool can support simultaneously (default value is 300). Expiration Time: Maximum time an unused thread remains in the pool (default value is 600,000 milliseconds). Sleep Time: Indicates how often the threads not used in the pool are checked to eliminate those that exceed the Expiration Time (default value is 300,000 milliseconds). Server Administration 142

154 Timeout: Maximum time allowed by the pool for execution of one of the threads (default value is 600,000 milliseconds). Checkout Time: Maximum time the pool waits for a free thread. If this time is exceeded, the system displays an error message (30,000 milliseconds by default). The configuration changes will become immediately effective Configuring the Cache As mentioned earlier, Virtual DataPort incorporates a system (called cache module) to store local copies of the data retrieved from the data sources, in a JDBC database. Virtual DataPort embeds an Apache Derby [DERBY] database that can be used to store the cache data, but it is also possible to store the cache in the following external DBMSs: Oracle 8i, 9i, 10g and 11g. MySQL 4.1 and 5.x (their drivers are not included). SQL Server 2000, 2005, 2008 and 2008R2. To activate the cache, access the Server Configuration Cache tab (see Figure 98), select the option Cache status on and fill in the following fields: DB adapter: Type of the database management system (DBMS) which will be used as cache. Virtual DataPort embeds an Apache Derby [DERBY] database which can be used to store the cache data by selecting the Embedded Derby Server check box. In this case, it is not required to fill in any other field. This embedded database can be configured to be accessible remotely by clicking on the Embedded Derby server configuration link (see below for more detail). Driver jar: Path to the.jar file of the JDBC driver. Alternatively, you can copy the.jar file to the directory $DENODO_HOME/extensions/thirdparty/lib and leave this field empty. After copying the file, you have to restart the Virtual DataPort Server and the Administration Tool so the driver can be loaded. Driver class: the name of the JAVA class of the JDBC driver. DB Uri: URI of the database used as cache. User name and password used to access the cache database. The user name should have read/write privileges on the specified database, including permissions to create and delete tables and indexes. If Test connection is selected (recommended), the system will connect to the source to obtain its manufacturer and version number. These data will be used to optimize access to the source depending on its specific characteristics. If this option is not selected, a generic configuration will be used. Click on Pool configuration to configure the parameters of the connections pool used to access the cache database (see section 5.3.1). The first time Virtual DataPort uses an external DBMS to store its cache data, it will generate the required tables and indexes automatically. Server Administration 143

155 By default, views do not store its data in the cache. Therefore, after enabling the cache module, you also need to activate the cache option in the views that require it (see section 5.6.2) Note: When using an external DBMS to store the cache data, it must be started before the Virtual DataPort server. Figure 98 Cache configuration It is also possible to consult and/or configure the following cache function parameters: Maintenance Period: specifies (in seconds) the execution period of the cache maintenance task. The maintenance task is in charge of deleting the expired entries of the cache. If the value provided is 0 or negative, the maintenance task will never be executed. Time To Live: specifies (in seconds) the default expiry time of cache inputs. If the cache is stored in the embedded database, some parameters can be configured by clicking on the Embedded Derby Server Configuration option: Enable Remote Access: it allows the embedded database to be accessed by external JDBC-compliant clients. This can be useful for auditing and monitoring purposes. Server port: this option allows changing the port where the embedded database runs. Likewise, to disable the cache simply select the Cache off option in the Cache panel of the Server Configuration tab. To invalidate partially or totally the data cached for a view, see sections 4.5.and The configuration changes will become immediately effective except when the cache DBMS is changed, in which case the server must be restarted.. A different cache configuration can be specified for each DataPort server Database (see section 9.3.2). Server Administration 144

156 8.1.4 Limiting the Number of Concurrent Requests It is possible to limit the number of requests that the DataPort server will accept concurrently. When that limit is reached, the new requests will be queued and executed according to their arrival order. Limiting the number of concurrent requests is useful in high load environments, since it avoids performance degradation issues when there is a peak load. To configure this behavior, it is necessary to access the Server Configuration tab and choose the option Concurrent Requests (see Figure 99). It is then possible to configure the following parameters: Activate/deactivate the limitation of concurrent requests. If deactivated, the server will try to execute all requests as they come. Maximum number of concurrent requests. The Virtual DataPort server will only accept this number of concurrent requests. Maximum number of requests on queue. Maximum number of queued requests waiting to be executed. New requests beyond this number will be discarded. Figure 99 Concurrent Requests Configuration Default Configuration of HTTP Proxy Some VDP DataSources can use http routes to access source data (see section 5.5.1). It is possible to configure the default proxy preferences for these connections by accessing the Server Configuration tab and choosing the HTTP Proxy option (see Figure 100). The configurable parameters are: Server Administration 145

157 Host. Name of the machine that acts as Proxy. Port. Port number in which the Proxy Server is being executed. Login. User identifier used to authenticate in the proxy. If the proxy does not require authentication, it must be left blank. Password. User access password. If the Proxy does not require authentication, it must be left blank. In the http route used for each DataSource, it is possible to specify a Proxy configuration different from the default one. Please see section for more information. Figure 100 HTTP Proxy Configuration Configuring the Wwapping to Disk Parameters To avoid memory overflows, Virtual DataPort can swap to disk the intermediate results generated during the execution of a query. It can also swap the intermediate results of the sorting operations (views with ORDER BY) To configure the swapping parameters of a server, go to the Server Configuration tab and select the Swapping option (see Figure 101). To enable or disable swapping on a global level, use the options Swapping status on (default option) or Swapping status off, respectively. Server Administration 146

158 Figure 101 Swapping Configuration It is also possible to configure the following swapping parameters: Maximum size of each intermediate result (megabytes). During the execution of a query, Virtual DataPort will swap to disk an intermediate result produced during the execution, when that result exceeds the maximum size specified in this parameter. As a general rule, the specified value should not be greater than one third of the memory available for the JAVA virtual machine on which the DataPort server is run. Maximum size of the blocks written to disk (kbytes). Where DataPort swaps an intermediate result, it will write to disk using blocks of, at most, the size specified. The general swapping configuration can also be specified for each Virtual DataPort database (see section 9.3.2). If the swap is globally enabled, it is possible to enable and configure it for one view (see section 5.6.3). It is also possible to configure it dynamically for a specific query using the VQL CONTEXT clause (see Context clause in the Advanced VQL Guide [VQL]). The configuration changes will become immediately effective Configuring Runtime Parameters for Stored Procedures Virtual DataPort allows executing stored procedures written in JAVA (see section 5.7 and the Stored Procedures section in Advanced VQL Guide [VQL]). Stored procedures can execute sentences on the DataPort server and process their results. The execution of those sentences is affected by several configuration parameters. Server Administration 147

159 To configure the runtime parameters for stored procedures, go to the Server Configuration tab and choose the Stored Procedures option (see Figure 102). Figure 102 Configuration of Stored Procedures runtime parameters It is possible to configure the following parameters: Query Timeout: Maximum time (in milliseconds) the stored procedure will wait for the termination of a sentence. If the value 0 is specified, the procedure will wait indefinitely until the sentence ends. Chunk Size: The results obtained by executing a sentence can be divided into blocks, so it is not needed to wait until a sentence ends to process the already obtained tuples. This parameter establishes the maximum number of results that a block can contain. If Virtual DataPort has obtained enough results to complete a block, they will be immediately returned to the stored procedure. The next results will be returned in new blocks. If this parameter is not specified (or receives the value 0), all the results of the query will be returned in a single block. Chunk Timeout: This parameter establishes the maximum time (in milliseconds) the server will wait before returning a new block. If this time is surpassed, Virtual DataPort will return the current block even if it still does not contain the number of results specified by the Chunk Size parameter. If the Chunk Timeout parameter is not specified (or receives the value 0), then all the results of the query will be returned in a single block when the sentence execution ends Configuring Default Internationalization The Virtual DataPort server internationalization configuration specifies aspects such as time zones, languages or currencies. For example, the default currency used in the Money-type values is specified by this property. Server Administration 148

160 Virtual DataPort includes typical internationalization configurations and also allows new configurations to be created ad hoc (see Creating New Internationalization Configurations in the Advanced VQL Guide [VQL]). A specific internationalization configuration can also be specified for each DataPort server Database (see section 9.3.2) Note: it is also possible to specify a different internationalization configuration for each base relation of the system (see section 5.6.1). The Administration Tool also has its own internationalization configuration that can be changed in the View - Preferences - Locale menu (see section 4.4.1) JMS Listeners Virtual DataPort can subscribe to a JMS server [JMS] to listen to VQL requests. Therefore, clients, instead of connecting to Virtual DataPort via JDBC, ODBC or a Web service, can send a request to a JMS server, which forwards it to Virtual DataPort. Then, the response is sent back to a queue or a topic of the JMS server, which forwards it to the client/s. E.g. a client sends a message such as SELECT * FROM internet_inc WHERE iinc_id=1 to the JMS server. The server will forward this to Virtual DataPort, which will send a response like: <?xml version="1.0" encoding="utf-8"?> <response> <item> <iinc_id>1.00</iinc_id> <summary>error in ADSL router</summary> <ttime>29-jun h 19m 41s</ttime> <taxid>b </taxid> <specific_field1>1</specific_field1> <specific_field2>1</specific_field2> </item> </response> Figure 103 Response message sent by a JMS listener If the request is a DML sentence such as ALTER VIEW incidences CACHE INVALIDATE, the response will be empty: <?xml version="1.0" encoding="utf-8"?> <response /> Figure 104 Response message to a DML query Follow these steps to create a JMS listener: 1. Go to the Server Configuration tab and click on JMS Listeners (see Figure 105) 2. Click on Add new listener 3. Fill in the form: Server Administration 149

161 a. Select On to enable the listener. This option is useful because later, we can disable the listener without deleting it. b. Select the database that the listener will connect to. JMS listeners, as JDBC clients, have to connect to a specific Virtual DataPort database where they will execute queries. c. Select the user name that Virtual DataPort will use to check if the listener has enough privileges to execute a query. E.g.: if we have two users: admin: is an administrator so it can access any view of any database. user1: is a normal user that only has READ privileges over the database samples. If we select the user admin, the listener can execute any query in the database selected in the box VDP database. However, if we select user1, we have to select the database samples because that user can only access that database. Besides, the CREATE / UPDATE / DELETE queries will fail because user1 only has READ privileges. 4. Destination is the name of the queue or topic that Virtual DataPort will subscribe to, waiting for requests. Depending on the vendor of the JMS server, we might have to create the JMS destination, or it will be created automatically when the listener tries to subscribe to it. 5. Select Queue or Topic. 6. Reply to is the name of the JMS queue or topic where the responses will be sent to. If empty, the responses will be sent to the destination specified in the Reply to field of the JMS request. 7. User name and Password are the credentials to connect to the JMS server. 8. JMS vendor. Select GENERIC if the vendor of the JMS server is not in the list and it can be accessed via JNDI. In this case, we have to provide the appropriate JNDI connection properties (appendix 12.8 contains a list of the required JNDI properties for the most popular JMS vendors) Note: the first time we create a JMS listener, we have to copy the client jars of the JMS vendor to the directory $DENODO_HOME/extensions/thirdparty/lib and restart the Virtual DataPort server. Section 12.8 contains a list of the client jars of the most popular JMS vendors. Server Administration 150

162 Figure 105 Creating a new JMS listener After creating the listener, we can see the list of existing JMS listeners (see Figure 106) Server Administration 151

163 Figure 106 List of existing JMS listeners By selecting the boxes beside the status of the listeners and clicking on Enabled selected and Disable selected, we can enable / disable JMS listeners. To delete a listener, click on its name to open its configuration and then, click on the button Delete. 8.2 EXPORTING / IMPORTING THE SERVER METADATA It is possible to export all server metadata to a VQL file, by using the Export option in the File menu of the Administration Tool (see Figure 107). It exports, among other things: The metadata from every database of the server. Users and their permissions. Configuration of JMS listeners. This option is available only for administrator users. Server Administration 152

164 Figure 107 Export dialog After exporting the metadata of a Virtual DataPort server to a VQL file, it can be imported back to the same server or another one. This is especially useful for migration and backup purposes. The Export dialog has the following options: Output file. Path of the new VQL file. Elements to export. Select the elements that will be included in the VQL file: o o o All Elements. Export all the elements of the server: data sources, views, Web services, users, JMS listeners, server settings, etc. Environment-specific elements. Only export the elements that depend on the server environment: data sources, users, JMS listeners and server settings. Non-specific elements. Only export elements that do not depend on a certain environment. In many organizations, installing a new application requires its validation in different environments (e.g. development, pre-production and production environments). Certain elements such as the paths or the authentication information used to access the data sources will normally be different in each environment. Usually, data sources are configured in each environment with the same name, but access the data source version available in each environment. Under these circumstances, it may be useful to export separately the elements that typically change from one environment to another and those that do not. For example, if users have created a new group of views in an environment and wish to pass it to another environment, they can export only the elements independent of the environment. The catalog elements considered Server Administration 153

165 dependent on the environment are data sources (see section 5.3), server properties (such as ports), users, and databases (see section 9). o Element. Select this option to export the metadata of a single element of the Virtual DataPort database that we are connected to. If the selected element type is View we can select the Export dependencies box to include in the output file, all the necessary elements to create this view (other views, types, data sources, etc) Replace / Drop elements. Select the action that a Virtual DataPort server will take when importing this VQL file and the server has elements (data sources, views, etc) with the same name and type as the ones that is importing: o Drop elements before creating them. Before creating a new element, it will delete, if exists, the element with that name and all the elements that depend on it. E.g. when a server imports a JDBC data source ds_jdbc_acme, first it will delete the existing JDBC data source ds_jdbc_acme and all the views that depend on this data source. This option is useful when we want to make sure that the imported data sources do not have more views that the ones contained in the VQL file. o o Replace existing elements. It will replace the existing elements with the elements from the file. E.g. when importing a LDAP data source ds_ldap_activedirectory and the server already has a LDAP data source with that name, the server will replace this data source with the one from the VQL file. However, it will not modify or delete the views that depend on the existing data source. Do not replace existing elements. It will keep the existing elements and ignore the ones from the file. E.g. if the server has a view internet_inc and the VQL file also contains a view with that name, the server will keep the existing view, instead of deleting it or replacing it. This option is useful when we need to add more elements to the server without affecting the existing ones. Include Jars. If selected, the output file will include the JAVA classes used to create stored procedures, CUSTOM data sources and custom functions. See section 8.3 and the Programming Extensions section in the Advanced VQL Guide [VQL]). Include ITPilot scanners. If selected, the output file will include the scanners used by WWW (Web) sources (see section and ITPilot User Guide [ITPILOT]). Include ITPilot custom components. If selected, the output file will include the ITPilot custom components used by the WWW data sources. Enter Single Mode during Import. If selected, the file generated will begin with the statement ENTER SINGLE USER MODE and finish with EXIT SINGLE USER MODE, so that Virtual DataPort switches to single user mode while importing this file (see Importing Metadata section in the Advanced VQL Guide [VQL]). Note: you are strongly advised to always import metadata in a Virtual DataPort server in single user mode. It is also possible to export the metadata from the active database only using the Export Database option in the File menu. In this case, it is only required to have privileges over the database to export. The metadata from a specific database can be also exported from its configuration screen (see section 9.3.2). Server Administration 154

166 Virtual DataPort also provides scripts to program the running of automatic backup copies (see section 11.3). There are three options to import the VQL file obtained in the export process: 1. Using the File Import dialog. 2. Using the VQL Shell tool (see section ) 3. Using the import / export scripts included in the utilities for importing and exporting metadata (see section 11.3) 8.3 IMPORTING EXTENSIONS The implementation of stored procedures (see section 5.7), custom functions and Custom wrappers (see section ) is done using JAVA code programming. See the Programming Extensions section in Advanced VQL Guide [VQL] for information and examples about how to create those elements. The File Extensions Jar Management option from the Administration Tool enables to add to the server new JAVA libraries (.jar files) that implement those elements. Server Administration 155

167 9 DATABASES, USERS AND ACCESS RIGHTS IN VIRTUAL DATAPORT This section describes various key concepts in the Virtual DataPort architecture. Section 9.1 describes the concept of databases in the context of a Virtual DataPort server. That is, logical groupings of datasources, views, base views, stored procedures and other elements in the server catalog. Section 9.2 describes the general concepts of user and access rights management in DataPort. Finally, section 9.3 describes how to manage this structure using the Administration Tool. The VQL commands for managing this structure are described in detail in the VQL Advanced Guide [VQL]. 9.1 DATABASES IN VIRTUAL DATAPORT A Virtual DataPort server can contain several databases (do not confuse with the possible external databases that can act as data sources of the system). A Virtual DataPort database represents a virtual schema comprised of a series of DataSources, views and stored procedures. Each database is independent of the other databases of the server and, as described in detail in the next section, different users can have different privileges for each database. When a Virtual DataPort server is installed, an initial database called admin is created. 9.2 USER AND ACCESS RIGHTS STRUCTURE IN VIRTUAL DATAPORT User Types Denodo Virtual DataPort distinguishes two types of users: Administrators. These can create, modify and delete databases in a DataPort server without any limitation. Likewise, they can also create, modify and delete users. When the server is installed, a default administrator user is created whose name is admin and whose password is also admin. This user can never be deleted. Normal users. These cannot create, modify or delete users. They cannot create or delete databases, although they can have connection, read, create or write privileges to one or several databases or to specific views contained therein Types of Access Rights Virtual DataPort access rights are applied to a specific user to delimit what tasks he/she is allowed to perform on databases, views and stored procedures. Access rights for a user can be applied globally to a database or specifically to a view or stored procedure in a specific database. Access rights to particular views and stored procedures are applied only if the user does not have the corresponding access right on a global level. Denodo Virtual DataPort supports the following types of global database access rights: Read access: Users with read access to a database on a global level can: Databases, Users and Access Rights in Virtual Dataport 156

168 o o o View the list of base relations, views and/or stored procedures of the database (corresponds to the VQL LIST command). If a user does not have read access to a database, but does have it for some of its views and/or stored procedures, the LIST command may be executed, but it will only display the group of views and procedures to which the user has read access. View the schema of any base relation, view or stored procedure of the database (corresponds to the VQL DESC command). Execute queries on any view and/or stored procedure of the database (corresponds to the VQL SELECT command). Creation access: Users with creation access to a database can: o Create DataSources, views, base relations and stored procedures in the database (corresponds to the VQL CREATE command). Write access: Users with write access to a database can execute the following tasks: (write access automatically implies read access too): o Delete any view, base relation and/or stored procedure of the database. Users can also delete any DataSource created by them, but not DataSources created by other users (corresponds to the VQL DROP command). o Modify any view, base relation and/or stored procedure from the database. Users can also alter any DataSource from their own databases, but not alter DataSources created by other users (corresponds to the VQL ALTER command) Connection access: If a user has this access to a database, then he/she can connect to it. This access right is useful, for example, if you wish to temporarily revoke a user s access to a database without having to alter the rest of his/her normal privileges. Denodo Virtual DataPort also supports individual privileges to specific views and stored procedures. The privileges that can be applied to a specific view and/or stored procedure of a database are: Read access: Users with this privilege on a view or stored procedure can: o View its schema (corresponds to the VQL DESC command). o Execute queries to it (corresponds to the VQL SELECT command). o Create new views that use it (only if the user has creation rights in the database). Corresponds to the VQL CREATE VIEW command. o If a user does not have read access to a database, but does have read access to some of its views and/or procedures, the LIST command may be executed, but only those components will be displayed. Write access: Having write access implies read access too. If a user has this privilege on a view or stored procedure, he/she may execute the following additional tasks on it: o Delete the component (corresponds to the VQL DROP command). o Edit the component (corresponds to the VQL ALTER command). Insertion access: Having this privilege on a view allows inserting tuples in it (this corresponds with the INSERT VQL statement). It is not applicable to stored procedures. Update access: Having this privilege on a view allows updating tuples from it (this corresponds with the UPDATE VQL statement). It is not applicable to stored procedures. Deletion access: Having this privilege on a view allows deleting tuples from it (this corresponds with the DELETE VQL statement). It is not applicable to stored procedures. Databases, Users and Access Rights in Virtual Dataport 157

169 9.3 GRAPHIC ADMINISTRATION OF DATABASES, USERS AND ACCESS RIGHTS To manage the databases, users and access rights of a Virtual DataPort server, access the administration tool in server administration mode, which implies doing it with an administrator-type user (in this case, it is not mandatory to specify a database in the server connection uri). When the server is installed, a default administrator user is created whose user name is admin and whose password is also admin. The following sections describe how to create/modify/delete databases and how to create/modify/delete users Creating Databases To create a new database, open the Database Management tab and press the new button. Figure 108 Creating a new database The tool will request the following data for the new database (see Figure 108): Name of the database. Description. Optional field to include a description of the database Configuration and Deletion of Databases To view the current list of databases of the Virtual DataPort server open the Database Management Databases tab. For each database, its name and description are displayed (see Figure 109). The value of the description field may be modified by pressing the edit button. Databases, Users and Access Rights in Virtual Dataport 158

170 Figure 109 Description of a database The available actions for a database are: Click the Assign privileges button to obtain a list of all Virtual DataPort users who are not administrators. For each user, you can specify whether the user has global connection, creation, read and/or write access rights to the database (see Figure 110). For more information about user privileges, read section The i18n, Swap and Cache buttons open the dialogs to change the internationalization settings (see section 8.1.8), swapping policies (see section 8.1.6) and cache settings (see section 8.1.3), respectively, for the database. In all three cases, if the Default Configuration box is selected, the Server Configuration will be used. By pressing the Export VQL button, all database metadata can be exported to a VQL file so that it can be easily recreated in another Virtual DataPort installation. Neither the users nor its access rights for the database are included in the export process. The database creation statement is not included either. We can also export all the metadata of the server, instead of just one database, using the File Export option (see section 8.2). Delete the database (button Delete ). Due to the delicacy of the operation, the Tool requests the user s password again) Important: when a database is deleted, all its components are eliminated: DataSources, views, base relations, etc. Databases, Users and Access Rights in Virtual Dataport 159

171 Figure 110 Assigning privileges on a database Creating Users To create a user, open the User Management tab and press the New User button. The tool then requests the following data for the new user (see Figure 111): User login. User password. It should be written twice so that it can be checked. Description. Optional field to include a description of the user. Type of user: Users can be of the type Administrator or Normal User (user without administration privileges). Authentication type. Users can be authenticated against DataPort ( Normal option) or against an LDAP server registered in DataPort (see section for information on how to register an LDAP server). The data to be completed is different, depending on the option chosen. o In the Normal case, a password will be requested for the user. It should be written twice so that it can be checked. o In the case of LDAP, the following data must be provided (see Figure 111): LDAP server to use. Select the database where the LDAP server required has been registered using the selectable Database. Once this has been done, select the LDAP data source using the selectable Datasource. LDAP user. The name of the user in the LDAP server. For example, the value 'cn=test,ou=people,dc=denodo,dc=com identifies the test user in an organizational unit People for the domain denodo.com. Databases, Users and Access Rights in Virtual Dataport 160

172 Figure 111 Creating a normal-type user NOTE: If a LDAP datasource is deleted on cascade (see section 5.3.2), then the users depending on it will also be deleted. This operation can only be executed by an administrator user. Databases, Users and Access Rights in Virtual Dataport 161

173 Figure 112 Creating an LDAP-type user Modifying and Deleting Users To see the current list of users of the Virtual DataPort server go to the User Management tab of the administration tool. For each user the login, description field and type are shown (administrator or normal user). See Figure 113. Databases, Users and Access Rights in Virtual Dataport 162

174 Figure 113 Description of a user The user description and password can be changed by pressing the edit button. If the user is not an Administrator, the option Assign privileges may be used to modify his/her privileges for the various DataPort databases (see section ). From the list of users screen, you can also delete the user by pressing the delete button (the system will request confirmation of the operation). The predefined admin administrator cannot be deleted Changing Active User Password The active user password can be modified using the Change password option on the application File menu. This option is available both in database administration mode and in server administration mode Modifying the Privileges of a User The privileges of the user for the different databases and views can be modified by using this option (this option is only available for users that are not administrators). Once the user s privileges edit screen has been accessed (see Figure 114), a list of the VDP server databases appears. For each database, a series of checkboxes allows assigning the user the desired global access rights on each database. Databases, Users and Access Rights in Virtual Dataport 163

175 Figure 114 Assigning privileges to a user at database level The privileges that can be assigned to a user in a database are: Connect. The user can connect to the database. Note: It is very important to grant this privilege to a user. Otherwise, the other privileges will be ignored. This privilege is useful to deny all the privileges of a user temporarily. Create. The user can create new elements in the database: data sources, views, publish web services, etc. Read. The user can access all the views and stored procedures of the database. Write. The user can modify / delete any view / stored procedures of the database. Write access implies read access. Admin. The user has the permission to connect, create, read and write to the database. Besides the user can: o Set the configuration parameters of the database: I18N, cache, swap, etc. o Edit the description of the database. o Grant / revoke privileges to normal users (not admin or database-admin users). It cannot grant the Admin privilege to other users. A user with this privilege cannot: o Create / delete users. o Change users password o Create / drop databases o Grant Admin privileges to other users. Databases, Users and Access Rights in Virtual Dataport 164

176 To configure the privileges on specific views and stored procedures of a database, click the edit link of the Advanced privileges column. This dialog contains a list of the views and stored procedures of the database (see Figure 115). For every view or stored procedure, there is a check box allowing to provide users with read, write, insert, update and/or delete access. Note: if you have assigned the user a certain privilege on a global level for the whole database, then the rights assigned to specific views and stored procedures will be ignored. Figure 115 Assigning privileges to a user at view and/or stored procedure level Databases, Users and Access Rights in Virtual Dataport 165

177 10 MONITORING THE VIRTUAL DATAPORT SERVER The Denodo Platform provides two different ways of monitoring the servers of the Platform: Virtual DataPort, Scheduler and ITPilot s browser pool and maintenance: 1. Using a Java Management Extensions (JMX) Agent such as jconsole. See section Using the Denodo Monitor (application included in the Denodo Platform). See section MONITORING WITH A JAVA MANAGEMENT EXTENSIONS (JMX) AGENT It is possible to access monitoring information of the server using the Java Management extensions (JMX) standard [JMX]. This information can be used to control the use of the server and audit the actions carried out on the data sources and/or the Virtual DataPort metadata. The available monitoring information is: General information on the server: number of active and inactive connections, memory usage, etc. Configuration and accesses to the server cache. Accesses to each data source. Accesses to the views and stored procedures. Transactions executed by the DataPort server. DataPort also generates different types of JMX notifications whenever: A DDL statatement is executed. E.g. ALTER TABLE, CREATE VIEW, etc. A DML statement is executed. E.g. SELECT, INSERT, etc. Transactions start or end. In JMX, the information exported is displayed as a set of objects known as MBeans. The MBeans exported by DataPort belong to the JMX domain com.denodo.vdb.management.mbeans. Any JMX client can be used to access the DataPort monitoring information and events. JMX jconsole (included with the Java Developer Kit 1.5 and above) and jmanage [JMANAGE] consoles, among others, have been successfully tested to access the information and events exported by the DataPort server. To use these consoles, it is usually needed to specify an URL to the server. For instance, if the DataPort server is running in the port 9999 in the local host, the URL would be: service:jmx:rmi:///jndi/rmi://localhost:9999/jmxrmi It is also possible to access the MBeans from an external application, using the JMX API [JMX]. Section provides information on how to use jconsole to access DataPort. The following subsections describe in further detail the information and events available in each of the following categories: general server information, general data source information, general cache information, information and events on DDL statements, information and events on DML statements and information and events on transactions. Monitoring the Virtual DataPort Server 166

178 Using JConsole jconsole is a server administration console compliant with the JMX standard that is distributed with the JDK 5 or higher. This section introduces the use of jconsole for accessing DataPort monitoring information and events. Refer to the JDK documentation for detailed information on the use of jconsole [JCONSOLE]. To run jconsole, execute: %JAVA_HOME%\bin\jconsole Note: jconsole is not included with the JRE, only with the JDK. In the agent connection window, select the Remote tab, enter the DataPort server data (machine, port, user name and password) and click on the Connect button. An administrator-type user must be used. Figure 116 Connection to DataPort from jconsole On connecting to the server, jconsole displays the default information for a monitored agent such as the server s uptime, number of threads or memory usage. Monitoring the Virtual DataPort Server 167

179 The specific DataPort server information is in the MBeans tab. On the left side of the screen, there s a list of the available MBeans. The MBeans contained in com.denodo.vdb.management.mbeans provide information about the Virtual DataPort server. VDBServerManagementInfo (see Figure 117) show, among other things, the number of active connections and the number of total connections since the server was started. The values of the properties that can be modified are displayed by jconsole in blue (e.g. the value of the CompleteManagementActive property is blue). Figure 117 Virtual DataPort MBeans The information and events provided by each MBean is described in the following sections. This section only shows an example to illustrate the use of jconsole with DataPort. Example: access to the information and events regarding active connections and DDL statements. On the left of Figure 117 are the MBeans containing the information on DataPort. All of these are uploaded to the domain com.denodo.vdb.management.mbeans. On the right of the figure are the properties of the MBean com.denodo.vdb.management.mbeans:type=vdbservermanagementinfo (see section Figure 120), which include the number of active connections and the number of total connections, since the server was started. The values of the properties that can be modified are displayed by jconsole in blue (e.g. the value of the CompleteManagementActive property is blue). Monitoring the Virtual DataPort Server 168

180 The information and events regarding DML statements (SELECT, INSERT, UPDATE, DELETE, etc.) are accessible via the MBeans uploaded to com.denodo.vdb.management.mbeans:type=requestsmanagementinfo (see section ). By clicking on this name, a list of the existing databases in the DataPort server will be displayed. By clicking on one of them, information on the DML statements run on this database will be accessed. For example, Figure 118 shows information on the queries run on the admin database. Information such as the total number of requests executed (TotalRequests property) or the number of active requests at that time (ActiveRequests property) is displayed. Click on the Refresh button to refresh the information on the attributes displayed on the current screen. Note, however, that each DDL request on DataPort appears as a new attribute. For jconsole to display new attributes (and, therefore, data corresponding to new requests), you must access the screen of any other MBean and then return to the original screen. This will ensure jconsole loads the new attributes. To view the data on each request, double click on its Value cell. The data about each statement includes the type of statement executed (SELECT, INSERT, etc.), the user who has run it, the time at which it was run and the exact statement run (see Figure 119). Figure 118 Attributes tab of the RequestsManagementInfo Mbean Monitoring the Virtual DataPort Server 169

181 Figure 119 Details of a query in jconsole Continuing with the example, the events related to DML statements from the admin database can be seen by clicking on the Notifications tab and then the Subscribe button. From then on, every time a DML statement is run in DataPort, a new notification will be displayed in jconsole. Double click on its UserData cell for details on the statement run (see Figure 120). Monitoring the Virtual DataPort Server 170

182 Figure 120 Details of a notification in jconsole General Information on the Server General information about the server is obtained through the Mbeans: com.denodo.vdb.management.mbeans.vdbservermanagementinfo com.denodo.vdb.management.mbeans.memorymanagementinfo com.denodo.mbeans.logmanagementinfo The properties of each one are described below: com.denodo.vdb.management.mbeans:type=vdbservermanagementinfo. o CompleteManagementActive: If true, complete monitoring mode is enabled. In that mode, an estimate of the current memory (TotalMemory property of the MBean MemoryManagementInfo) and the memory peak occupied by a query (MaxMemory property of the MBean MemoryManagementInfo) is calculated. The monitoring of these parameters is costly and, therefore, this mode should only be enabled when necessary. This property can be modified by the user and can take the values true or false. o ServerName: Virtual DataPort server URI (e.g. //localhost:9999/vqlserver) o TotalConnections: Total number of connections established with the server, since it was started. Monitoring the Virtual DataPort Server 171

183 o ActiveConnections. Number of currently active connections. com.denodo.vdb.management.mbeans:type=memorymanagementinfo. The attributes of this MBean are only calculated if complete monitoring mode is enabled (see above). o TotalMemory. Estimate of the total memory occupied by the server. o MaxMemory. Estimate of the memory peak reached by the server since its start. o gc (). Operation that executes the garbage collector of the JAVA virtual machine of the Virtual DataPort server. com.denodo.mbean.logmanagementinfo. It has operations to obtain and change the appenders of Log4J [LOG4J] o getloglevel(<logger name>).obtains the log level of a logger. I.e: com.denodo o setloglevel(<logger name>, { DEBUG INFO WARN ERROR }) Sets the log level of a logger General Information on Data Sources General information about data sources is obtained through the MBean: com.denodo.vdb.management.mbeans:type=datasourcemanagementinfo,name=gl obaldatasourcemanagementinfo This MBean has the following attributes: TotalDataSourceCount. Number of datasources TotalActiveRequestCount. Number of current requests to a datasource. For every type of datasource there is an attribute with the number of datasources and the number of active request to that type of source. ITPMaintenanceDataSourceCount. Number of Web (WWW) sources that have its maintenance option enabled. ITPMaintenanceActiveRequestsCount. Number of current requests to Web sources that have its maintenance option enabled. Each VDP database has its own MBean with the same attributes, but only considers data sources that belong to that database: com.denodo.vdb.management.mbeans:type=datasourcemanagementinfo,databas ename=<vdp database name>,aux=localdatasourcemanagementinfo Besides, each datasource has its own MBean: com.denodo.vdb.management.mbeans:type=datasourcemanagementinfo, datasourcetype=<dstype>,datasourcename=<dsname>. where <dstype> is the type of data source (JDBC, ODBC, WS, XML, etc.) and <dsname> is the name of the data source. These MBeans have the following attributes: DatabaseName. Name of the data source database. DataSourceName. Name of the data source. DataSourceType. Type of the data source: JDBC, ODBC, XML, WS (Web services), DF (delimited files), ARN (Aracne), GS (Google Mini), CUSTOM, LDAP, ITP (Web). NumRequests. Number of requests made on the data source since the start of the server. ActiveRequests. Number of active requests to the data source. Monitoring the Virtual DataPort Server 172

184 MaxActive, NumActive and NumIdle (only for JDBC and ODBC sources). Provide information about the pool of connections with the database. o MaxActive: maximum number of connections in the pool. o NumActive: number of connections established with the source that are being used to execute a query. o NumIdle: number of idle connections established with the source. ismaintenanceenabled (only for Web sources). True if the maintenance option is enabled Information on the Cache General information on the server cache is obtained through the MBeans: com.denodo.vdb.management.mbeans:type=cache,databasename=<dbname >,subtype=databasecache, where <dbname> specifies the name of a DataPort database, and com.denodo.vdb.management.mbeans:type=cache,databasename=<dbname >,subtype=viewcache,viewname=<vname>, where <dbname> specifies the name of a DataPort database and <vname> specifies the name of a view of said database. Below is a description of the properties of each one of these two subtypes: subtype=databasecache. The following properties are given for each DataPort database: o DatabaseName. Name of the database. o CacheStatus. Status of the cache: ON or OFF o UserName. Name of the user in the data source associated with the database used as cache. o UserPwd. Password of the user in the data source associated with the database cache. o DriverClassName. Class of the JDBC driver in the data source associated with the database used as cache. o LastCacheAccessStatus: Indicates whether the last access to the cache database was successful (true) or not (false). o CustomDataSource: (read only) This indicates whether the database used as cache uses (true) or not (false) a different data source to that used by default by the DataPort server. o DBURI: Connection URI to the database used as cache. o iscustomdatasource(). Operation that returns true if the database uses a different cache configuration that the DataPort server. False otherwise. subtype=viewcache. The following properties are given for each view (viewname) of every database (databasename): o DatabaseName. Name of the DataPort database to which the view belongs. o ViewName. Name of the view. o CacheStatus. Status of the view cache (ON/OFF/POST). o TTLInCache. Lifetime of the data in the view cache. o LastAccess. Moment of the last access to the view cache. o LastRefresh. Moment in which new tuples were last written in the view cache. o RefreshFailuresCount. Number of failures when writing new tuples in the view cache Information and Events on Catalog Access (DDL Statements) The information and events about the DDL statements is obtained through the MBean: Monitoring the Virtual DataPort Server 173

185 com.denodo.vdb.management.mbeans:type=catalogmanagementinfo, databasename=<dbname> where <dbname> specifies the name of a DataPort database. The MBeans of this type offer a single property (read only) called DatabaseName that indicates the name of the database. On subscribing to the MBean of a database, a notification is received every time one of the following events occurs: A statement affecting the metadata of the database is executed (e.g. a new view is created, a view definition is modified, the privileges of a user are changed, etc). A transaction starts. A transaction ends. The following fields appear in each notification: Timestamp. Moment in which the notification of the statement is generated in the JMX server. Type. Type of the transaction: start of a transaction ( starttransaction ), end of a transaction ( endtransaction ) or execution of a DDL statement ( newevent ). UserData. Compound element. Its subproperties depend on the type of notification: o Statement executed: DatabaseName: Name of the DataPort database on which the statement is run. EventID: ID of the notification generated by DataPort. Time: Moment at which the statement is run according to the DataPort server. TransactionID: ID of the transaction within which the statement is run. Type: Name of the executed operation. UserName: ID of the user running the statement. VQLQuery: VQL code for the statement. o o Transaction start: Autostarted: Indicates whether the transaction has been explicitly created by the user. All statements modifying the DataPort catalog must be run within a transaction. Therefore, if the user has not created it, DataPort will create it itself. DatabaseName: Name of the DataPort database on which the transaction is started. EventID: ID of the notification generated by DataPort. The IDs are allocated using a counter. Time: Moment at which the transaction starts according to the DataPort server. TransactionID: ID assigned to the starting transaction. UserName: ID of the user starting the transaction. Transaction end: DatabaseName: Name of the DataPort database on which the transaction is run. EventID: ID of the notification generated by DataPort. The IDs are allocated using a counter. Status: Completion status of the transaction (COMMIT or ROLLBACK). Time: Moment at which the transaction finishes according to the DataPort server. TransactionID: ID assigned to the ending transaction. UserName: ID of the user running the transaction. SeqNum: Sequence number generated by DataPort for the notification. Normally, the sequence numbers are assigned using a counter. Message: Monitoring the Virtual DataPort Server 174

186 o If the execution of a statement is notified: Processed the <event eventid> in transaction <transactionid>. o If the start of a transaction is notified: Started the transaction <transactionid>. o If the end of a transaction is notified: Finished the transaction <transactionid>. Event: This will be javax.management.notification[com.denodo.vdb.management.mbeans:t ype=catalogmanagementinfo,databasename=<dbname>][type=<eventtype >][message=<eventmessage>], where <dbname> is the name of the DataPort database, <eventtype> is starttransaction, endtransaction or newevent and <eventmessage> takes the same value as in the Message property. Source: MBean name Information and Events on the Running of DML Statements The information and events about the executed DML statements can be obtained through the MBean: com.denodo.vdb.management.mbeans:type= RequestsManagementInfo,databaseName=<dbname>, where <dbname> specifies the name of a DataPort database. The following properties are given for each database: DatabaseName. Name of the DataPort database. MaxRequests. Maximum number of requests exported at the same time in the MBean. Each request appears as an attribute of the form Request<i>. TotalRequests. Total number of requests recorded since the start of the server. ActiveRequests. Number of active requests. Request<i>. Compound element. Its subproperties are: o Cache. True if the query has accessed the cache during its execution. False otherwise. o Completed. True if the query finished correctly. False otherwise. o DatabaseName. Name of the DataPort database on which the statement is executed. o EndTime. Moment at which the statement finished its execution. o Identifier. Transaction identifier. o MaxMemory. Estimate of the memory peak consumed by the query. Only available if the CompleteManagementActive of the MBean com.denodo.vdb.management.mbeans:type=vdbservermanagementi nfo is enabled. o Memory. Estimate of the memory consumed by the statement. Only available if the attribute CompleteManagementActive of the MBean com.denodo.vdb.management.mbeans:type=vdbservermanagementi nfo is enabled. o NumRows. Number of rows returned by the query. o RequestType. This indicates the type of statement. It can take the following values: SELECT BASE VIEW, SELECT VIEW, QUERY WRAPPER, CALL STOREDPROCEDURE, INSERT and UPDATE. o SessionId. Id of the session that this request belongs to. o StartTime. Moment at which the statement started its execution. o State. It can take the following values: OK, ERROR or PROCESSING. o Swap. True if the server swapped the intermediate results to disk during the execution of the query. False otherwise. Monitoring the Virtual DataPort Server 175

187 o o o UserName. ID of the user running the statement. VQLQuery. VQL code for the statement. WaitingTime. Time that request was waiting in the queue before being processed. Furthermore, the MBeans of this type also allows subscribing to notifications. Every time a DML statement is run on the specified database, two notifications are received: one, indicating the beginning of the request and the other indicating the end. Timestamp. Moment in which the notification is generated in the JMX server. Type. This type of notification includes the string startrequest or endrequest as the type indicator. UserData. Compound element. Its subproperties are the same as those of the Request<i> property described above. SeqNum. Sequence number generated by DataPort for the notification. Message. In this type of notification, it takes the value Started the request or Finished the request. Event. This will be javax.management.notification[source=com.denodo.vdb.management.m beans:type=requestsmanagementinfo,databasename=<dbname>][type=en drequest][message=finished the request '<i>'], where <dbname> is the name of the DataPort database and <i> is the statement identifier. Source. MBean name Information and Events on Transactions The information and events about the transactions can be obtained through the MBean: com.denodo.vdb.management.mbeans:type= TransactionsManagementInfo,databaseName=<dbname>, where <dbname> specifies the name of a DataPort database. The following properties are given for each database: DatabaseName. Name of the DataPort database. MaxTransactions. Maximum number of transactions exported at the same time in the MBean. Each transaction appears as an attribute in the form Transaction<i>. TotalTransactions. Total number of executed transactions since the start of the server. ActiveTransactions. Number of currently active requests. Transaction <i>. Compound element. Its subproperties are: o Autostarted. Indicates whether the transaction has been explicitly created by the user. All statements modifying the DataPort catalog must be run within a transaction. Therefore, if the user has not created it, DataPort will create it itself and, in this case, the value of this property will be false. o DatabaseName. Name of the DataPort database on which the transaction is run. o EndTime. Moment at which the transaction ended. o Identifier. Transaction ID. o StartTime. Moment at which the transaction started. o State. This indicates the status with which the transaction ended. It can take the following values: ROLLBACK or COMMIT. o UserName. ID of the user running the transaction. Monitoring the Virtual DataPort Server 176

188 It is possible to subscribe to the events of the TransactionsManagementInfo MBean. In that case, every time a transaction starts or ends on the specified database, a notification with the following data is received: Timestamp. Moment in which the notification is generated in the JMX server. Type. This type of notification takes the string starttransaction or endtransaction as the type indicator, depending on whether the notification indicates the start or the end of a transaction. UserData. Compound element. Its subproperties are the same as those of the Transaction<i> property described above. SeqNum. Sequence number generated by DataPort for the notification. Normally, the sequence numbers are assigned using a counter. Message. If the notification indicates the start of a transaction, its value is Started the transaction. If it indicates the end of a transaction, its value is Finished the transaction. Event. This will be javax.management.notification[source=com.denodo.vdb.management.m beans:type=transactionsmanagementinfo,databasename=<dbname>][typ e=<type>][message=<msg>], where <dbname> is the name of the DataPort database, <type> is the type of notification ( starttransaction or endtransaction ) and <msg> is the notification message ( Started the transaction or Started the transaction ). Source. MBean name DENODO MONITOR The Denodo Monitor is a tool included in the Denodo Platform that logs several parameters of the Denodo servers: Virtual DataPort, Aracne and Aracne Index [ARCN], ITPilot s browser pool and maintenance [ITPILOT] and Scheduler [SCHED]. Denodo Monitor includes several types of monitors which obtain information from each server. There is a monitor to log the active processes and connections of the local computer, memory resources used by each Java Virtual Machine, etc. The Denodo Monitor is located in the directory $DENODO_HOME/tools/monitor, compressed in a zip file. To execute it, decompress this file and execute one of the following commands: a) denodomonitor_startup.sh/bat Launches de Denodo Monitor and keeps it running indefinitely. Periodically, it obtains information from all the monitors and logs it into text files located in the denodo-monitor\logs directory. b) denodomonitor_startup.sh/bat -s Makes a single execution of all the monitors, prints the results in the logs and exits. Note: in this mode, the VDP queries monitor is not launched, even if it is configured to do so. c) denodomonitor_startup.sh/bat -h { vdp browserpool arn arnindex maintenance scheduler } It obtains a heap dump of one of the Denodo servers by connecting to the JVM of that server via JMX and requests it to make a memory dump. The memory dump will be stored in the server machine s temporal directory. Note: in Unix environments, give execute permissions to the script: chmod +x bin/denodomonitor_startup.sh There are three different types of monitors : Monitoring the Virtual DataPort Server 177

189 Local monitors. They gather information about the machine where the Denodo Monitor is running. Server monitors. They obtain monitoring information by connecting to a server via JMX [JMX]. Thus, they can obtain this information from remote servers. VDP Queries monitor. It logs several parameters of all the VQL queries received by a VDP server. It also obtains this information via JMX, so it can also monitor remote servers Local Monitors Local monitors execute commands in the local computer to obtain certain information Processes Monitor This monitor logs information about all the running processes in the local computer. In Windows OS, it logs the output of the command tasklist.exe and in Linux, the output of ps. The log file is stored in conf/processes.log Sockets Monitor This monitor logs the information of the active connections of the local computer by executing the command "netstat". The log file is stored in conf/sockets.log Server Monitors These monitors connect to a Denodo server via JMX. The difference with the local monitors is that local monitors can only obtain data of the machine they are executed in, while server monitors can obtain information remotely Resources Monitor This monitor logs several parameters of the memory footprint of the server. The log file is stored in conf/<server name>-resources.log Every time the information about resources is logged, the following information is collected: Memory information: various parameters of the memory usage of the JVM that executes the monitored server: o Perm Gen o Survivor Space o Tenured Gen o Eden Space o Code Cache o Heap Memory Usage o Non Heap Memory Usage Classes information: o Loaded Class Count. Number of currently loaded classes. o Total Loaded Class Count Total. Total number of loaded classes. o Thread Count. Number of threads. o Peak Thread Count. Maximum number of threads. VDP Information (only for monitors connected to a Virtual DataPort server) Monitoring the Virtual DataPort Server 178

190 o o o o o o VDP Total Conn. Number of connections opened during the interval. VDP Active Conn. Number of currently opened connections. VDP Active Requests. Number of request that are currently being executed. VDP Waiting Requests. Number of enqueued requests. VDP Total Mem. Estimate of the total memory occupied by the server. VDP Max Mem. Estimate of the memory peak reached by the server since its start Threads Monitor This monitor logs basic information and CPU usage of every thread of the server. It is very useful to identify VQL queries that consume too much CPU time. The log file is stored in conf/<application name>-threads.log Virtual DataPort Queries Monitor The Virtual DataPort Queries monitor is specific to Virtual DataPort. It subscribes to the RequestsManagementInfo MBean (see section ), which sends a notification every time Virtual DataPort receives a query. The monitor stores the information contained in these notifications, in $DENODO_HOME/tools/monitor/conf/vdp-queries.log Configuring the Denodo Monitor The configuration file of the Denodo Monitor is $DENODO_HOME/tools/monitor/conf/ConfigurationParameters.properties. Almost all the properties in this configuration file are self-explanatory, so we are going to review only the most important ones: monitors.local. Comma-separated list of local active monitors : processes ( ) and sockets ( ). monitors.remote. Comma-separated list of servers to monitor: vdp, browserpool, arn, arnindex, maintenance or scheduler. {arn arnindex browserpool maintenance scheduler vdp}.monitors. Properties that indicate which remote monitors will be active for each server. The possible values for these properties are: vdpqueries (only for the property vdp), threads and resources. {processes sockets resources threads}.interval. Number of seconds between two JMX requests issued by that monitor. vdpqueries.snmptrapagent.enable. If true, the monitor sends SNMP traps with the information received by the Queries monitor. To use it, uncomment and configure the properties vdpqueries.snmptrapagent. The properties.jmx.password store the password to connect to the server of that property. These passwords can be stored encrypted. To do this, set the value of the property.jmx.password.encrypted to true if the value of.jmx.password is encrypted. For example, by default the Virtual DataPort password is admin. If you changed it and you want to store it in this configuration file encrypted, do the following: 1. Execute the script $DENODO_HOME\tools\monitor\denodo-monitor\bin\encrypt_password <new password> Monitoring the Virtual DataPort Server 179

191 The output of this script is the encrypted password that you have to put in the configuration file of the Denodo Monitor. 2. Open the file $DENODO_HOME\tools\monitor\denodomonitor\conf\ConfigurationParameters.properties 3. Set the value of the property vdp.jmx.password to the encrypted password you obtained in step Set the value of the property vdp.jmx.password.encrypted to true. Monitoring the Virtual DataPort Server 180

192 11 CLUSTER ARCHITECTURES / SERVER BACKUP Virtual DataPort can operate within a cluster architecture to provide high availability and load balancing. Typically, clustering architectures for DataPort operate as follows: 1. An external load-balancing system distributes the requests received among a pool of Virtual DataPort servers. 2. The load-balancing system detects when a server is not responding (using the so-called server health check ). In this case, the load is distributed among the remaining servers. 3. The clustering system detects when the server is available again (using the health check ) and adds it to the pool. The following clustering solutions have been tested with Virtual DataPort: Linux Virtual Server architecture [LVS], using Keepalive [KALV] to check whether a server is responding at any given time. Windows 2000 Network Load Balancing Tool [WNLB]. This is the clustering solution included in the Windows 2000 server family. Virtual DataPort provides some features to make cluster architecture configuration easier: It provides a PING script to check the status of a DataPort server. Although solutions such as Keepalived provide their own default implementations for health-checking an application, the PING script provided by DataPort detects failure situations that would not be detected by other solutions. It provides a utility to automatically replicate the metadata and the configuration of a DataPort server in a pool of other DataPort servers. This utility can also be used to schedule backup copies of the metadata of a DataPort server. The following sections describe how to: 1. Install a set of tools that help building a cluster with several Virtual DataPort servers (section 11.1). 2. Use the PING script (section 11.2) 3. Replicate the metadata of a Virtual DataPort server, to a group of servers (section 11.3) TOOLS INSTALLATION The tools are located in the path %DENODO_HOME%\tools\db\denodo-db-tools.zip. To use them, decompress this file USE OF THE PING SCRIPT The script ping is located in the directory %DENODO_HOME%\tools\db\denodo-db-tools\bin. The syntax of this script is: Cluster Architectures / Server Backup 181

193 ping [-t timeout] [-v] [//]host[:port], where host and port are the machine and port where the Virtual DataPort server runs (port 9999 is assumed by default, if it is not specified) and timeout is the maximum time (in milliseconds) to wait for the server response. Option v shows via the standard output the status of the server and the time taken to receive a response from it. The ping script returns 0 if the status check is successful. Otherwise, it returns another value. An example of running the ping command is shown below: ping -t v //localhost:9999/ This command checks the status of a Virtual DataPort server running in the local machine, listening on port The script waits at most 5 seconds for the server s response. The result of the check is displayed by the standard output USE OF THE IMPORT/EXPORT SCRIPTS FOR BACKUP AND/OR REPLICATION The import and export scripts are part of the Denodo Tools toolkit ($DENODO_HOME/tools/db/denodo-dbtools.zip) The export script exports the metadata of a Virtual DataPort server to a file. We can export all the metadata of the server and its configuration or just the metadata of a specified database. The output of the script is equivalent to the Export and Export database output of the Administration Tool (see section 8.2). The syntax of the script is: export l <login> -p <password> u <host>[:<port>][/<database name>] [-i] f <outputfilename> s [-P <property name>=<property value>]* where: --login <login>. Login name used to access to the Virtual DataPort server. The user must have sufficient access rights to export data. --password password. Password used to access to the server. --uri <host>:[<port>][/<database name>]. URI of the server (e.g: localhost:9999/admin). If no database is specified in the URI, the script exports all the metadata of the server. Otherwise, it only exports the metadata of that database. If no port is specified, the script tries to connect to the port i. Export only WWW wrappers. This option is useful to export content from an ITPilot server, or to export ITPilot wrappers [ITPILOT] from a Virtual DataPort database so that they can be later imported to another ITPilot server. Note: Only use this option if the URI contains a database name. f outputfilename. Name of the target file. -s. If present, Virtual DataPort servers will switch to single user mode while importing the file outputfilename. -P name=value. Set of settings (explained in section 8.2) that control the export process. You can specify one or more properties following the <property name> = {yes no} format. The available properties are: includejars includeenvspecificelements Cluster Architectures / Server Backup 182

194 includenonenvspecificelements includedependencies includescanners includecustomcomponents replaceexistingelements dropelements Example: export -l admin -p admin -u //localhost:9999/admin -f serverexport.vql P includejars=yes includescanners=yes includeenvspecificelements=no includenonenvspecificelements=yes The script exports all the metadata of the Virtual DataPort server running in localhost, port 9999, using the credentials admin / admin. The result of the export is saved to the file serverexport.vql. The import script imports Virtual DataPort metadata that was obtained using the export utility, to a list of Virtual DataPort servers. The URI of these servers can be provided by command line or readed from a text file. Thus, it is possible to restore a backup copy made with export and to replicate the metadata of a server into a cluster of servers. The syntax of the script is: import [-u] -f path-to-file [-s servers-file -l server (-l server)*] where: -u (--singleuser). The server switches to single user mode (see Advanced VQL Guide [VQL]) while importing the metadata of the file. Note: you are strongly advised to always import metadata in single user mode. -f path-to-file. Path to the file containing the metadata to be imported. -l server. URI of the server to which the metadata is imported. The syntax of the URI is host[:port]/database?user@password. If port is missing, the script assumes that the port is s servers-file. Instead of using the parameter l, we can provide the path to a file containing URIs of the servers to which the metadata will be imported. Each URI has to be indicated in a separate line, with the same syntax of the parameter l. Example: import -f export.vql -l localhost/admin?admin@admin1 -l host2:9099/admin?admin@admin2 The script imports the metadata contained in export.vql to the servers running in the local machine, port 9999 and in host host2, port For both servers, authentication is made against the database admin, and login admin. The passwords for the servers are admin1 and admin2, respectively. import -f export.vql -s servers.conf The script imports the metadata contained in export.vql and reads the target URIs from the file servers.conf. Cluster Architectures / Server Backup 183

195 11.4 CONFIGURING SEVERAL INSTANCES OF A VIRTUAL DATAPORT SERVER It is possible to launch two or more instances of a Virtual DataPort Server at the same time. Each instance is executed independently, which means that each one is a different process of the operating system. However, all of them share the same metadata (data sources, views, etc) and the same embedded Web container. When executing two or more instances of the same Server, one of the instances is the Master and the others are the Slaves. The Master is the only one that can modify the metadata of the Server (create/edit/remove data sources, views, etc). This instance has to be launched before the Slave instances How to Configure the Slave instances Let us say that we want to add two new Slave instances called aux_1 and aux_2. To do this, follow these steps: 1. Edit the file $DENODO_HOME/conf/vdp/VDBConfiguration.properties and add the following properties: vdp.instances=aux_1,aux_2 com.denodo.vdb.vdbinterface.server.vdbmanagerimpl.aux_1.registryurl=localhost com.denodo.vdb.vdbinterface.server.vdbmanagerimpl.aux_1.registryport=19999 com.denodo.vdb.vdbinterface.server.vdbmanagerimpl.aux_1.shutdownport=19998 com.denodo.vdb.vdbinterface.server.vdbmanagerimpl.aux_1.factoryport=19997 com.denodo.vdb.vdbinterface.server.vdbmanagerimpl.aux_1.odbcport=19996 com.denodo.vdb.vdbinterface.server.vdbmanagerimpl.aux_2.registryurl=localhost com.denodo.vdb.vdbinterface.server.vdbmanagerimpl.aux_2.registryport=29999 com.denodo.vdb.vdbinterface.server.vdbmanagerimpl.aux_2.shutdownport=29998 com.denodo.vdb.vdbinterface.server.vdbmanagerimpl.aux_2.factoryport=29997 com.denodo.vdb.vdbinterface.server.vdbmanagerimpl.aux_2.odbcport=29996 Note that for each element of the property vdp.instances, you have to add the following properties: com.denodo.vdb.vdbinterface.server.vdbmanagerimpl.<name of the instance>.registryurl com.denodo.vdb.vdbinterface.server.vdbmanagerimpl.<name of the instance>.registryport com.denodo.vdb.vdbinterface.server.vdbmanagerimpl.<name of the instance>.shutdownport com.denodo.vdb.vdbinterface.server.vdbmanagerimpl.<name of the instance>.factoryport com.denodo.vdb.vdbinterface.server.vdbmanagerimpl.<name of the instance>.odbcport 2. Optionally, you can configure the options of the JVM for each new instance. To do this, add the following properties to the file $DENODO_HOME/conf/vdp/VDBConfiguration.properties. java.env.aux_1.denodo_opts_start=-mx2048m -XX\:MaxPermSize\=256m java.env.aux_1.denodo_opts_stop=-mx64m java.env.aux_2.denodo_opts_start=-mx2048m -XX\:MaxPermSize\=256m java.env.aux_2.denodo_opts_stop=-mx64m Note that for each element of the property vdp.instances, you have to add the following properties: java.env.<name of the instance>.denodo_opts_start= java.env.<name of the instance>.denodo_opts_stop= The property DENODO_OPTS_START has the JVM options that the instance will launched with. The property DENODO_OPTS_STOP has the JVM options that the script that stops the Server will be launched with. Cluster Architectures / Server Backup 184

196 Note: the characters : and = have to be escaped with the character \. 3. Execute the script $DENODO_HOME/bin/regenerateInstanceScripts. This scripts will create the directory $DENODO_HOME/bin/instances, which will contain the scripts to launch the Slave instances of the Master Server. For each instance, there will be: a. A script to launch the instance: vqlserver_<name of the instance>_startup b. A script to stop the instance: vqlserver_<name of the instance>_shutdown c. If running the Server in Windows, a script to create a new Windows service, remove it, launch and stop the service: vdpservice_<name of the instance>.bat 4. Launch the Master instance before the Slave instances. You can do it from the Denodo Control Center or executing the script $DENODO_HOME/bin/vqlserver_startup. 5. Launch the Slave instances by executing the following scripts: $DENODO_HOME/bin/instances/vqlserver_aux_1_startup $DENODO_HOME/bin/instances/vqlserver_aux_2_startup Now you can connect to any of the three instances (the Master or the Slaves) to query views. However, you have to connect to the Master instance to modify the metadata of the Server. That is, to create/edit/delete data sources, views, etc. Cluster Architectures / Server Backup 185

197 12 APPENDICES 12.1 JDBC DRIVERS The following table contains the JDBC drivers that have been successfully tested with Virtual DataPort 4.6. For each DBMS, the table indicates: Class. Class name that must be specified when importing a data source using that driver. URI. URI format used by the driver. DBMS Class URI DB2 8.2 DB2 9 com.ibm.db2.jcc.db2driver jdbc:db2://<hostname>:<port>/<databasename> Informix 7 com.informix.jdbc.ifxdriver jdbc:informixsqli://<hostname>:<port>/<databasename>:informix server=<server> MySQL MySQL com.mysql.jdbc.driver jdbc:mysql://<hostname>:<port>/<databasename> MySQL 5.x Oracle 8i Oracle 9i Oracle 10g Oracle 11g Oracle EBS 12 oracle.jdbc.oracledriver jdbc:oracle:<protocol>:@<hostname>:<port>:<datab asename> Protocols: thin (recommended), oci, oci8, kprb PostgreSQL PostgreSQL PostgreSQL 8 PostgreSQL 9 SQL Server 2000 SQL Server 2005 SQL Server 2008 SQL Server 2008R2 SQL Server 2000 SQL Server 2005 SQL Server 2008 SQL Server 2008R2 Sybase Adaptive Server Enterprise 12.5B Sybase Adaptive Server Enterprise 15 Sybase Adaptive Server Enterprise 12.5B Sybase Adaptive Server Enterprise 15 org.postgresql.driver net.sourceforge.jtds.jdbc.driver com.microsoft.jdbc.sqlserver.sqls erverdriver net.sourceforge.jtds.jdbc.driver com.sybase.jdbc3.jdbc.sybdriver jdbc:postgresql://<hostname>:<port>/<databasena me> jdbc:jtds:sqlserver://<hostname>:<port>/<database Name> jdbc:microsoft:sqlserver://<hostname>;databasena me=<databasename> jdbc:jtds:sybase://<hostname>:<port>/<databasena me> jdbc :sybase:tds:/<hostname>/:/<port>/<database> Table 9 JDBC Drivers tested with Virtual DataPort Note: The following drivers are not included with Virtual DataPort, although they can be downloaded from the websites of those companies: IBM DB2, MySQL, Microsoft SQL Server and Sybase. Any JDBC driver should work correctly with Virtual DataPort, but the ones listed in Table 9 have been successfully tested. Appendices 186

198 To use a JDBC driver not included with DataPort, copy the jar file of the driver to $DENODO_HOME/extensions/thirdparty/lib Alternatively, when creating a new JDBC data source, you can directly specify the.jar file containing the desired driver (see section 5.3.1). Appendices 187

199 12.2 MULTIDIMENSIONAL TO RELATIONAL MAPPING Virtual DataPort can extract data from multidimensional databases by executing MDX queries. MDX (Multidimensional Expressions) is a query language for multidimensional databases, as SQL is a query language for relational databases. The result of an MDX query is formed by axes, which are similar to the dimensions of a multidimensional structure. Virtual DataPort maps these axes into a relational structure by creating one field for each value of the ON COLUMNS axis and one field for each dimension of the ON ROWS axis. If a dimension of the ON ROWS axis is hierarchical, there will be a column for each level of this dimension. Note: Virtual DataPort only supports MDX queries with the ONROWS and ONCOLUMNS axes. Figure 121 Relational star schema SELECT [Measures].MEMBERS ON COLUMNS, NON EMPTY CROSSJOIN ([Region].MEMBERS, [Time].MEMBERS) ON ROWS FROM [$Sales] Figure 122 MDX query for Visual result mapping (1) Table 10 contains the result of executing a multidimensional base view created with the MDX query of Figure 122 and Visual result mapping. RegionLevel00 RegionLevel01 RegionLevel02 TimeLevel00 TimeLevel01 units_sold units_returned All Region <null> <null> All Year <null> All Region <null> <null> All Year All Region <null> <null> All Year All Region Europe <null> All Year <null> All Region Europe <null> All Year All Region Europe <null> All Year All Region Europe Spain All Year <null> All Region Europe Spain All Year All Region Europe Spain All Year All Region Europe France All Year <null> All Region Europe France All Year All Region Europe France All Year All Region America <null> All Year <null> Appendices 188

200 All Region America <null> All Year All Region America <null> All Year All Region America U.S.A All Year <null> All Region America U.S.A All Year All Region America U.S.A All Year All Region America Brazil All Year <null> All Region America Brazil All Year All Region America Brazil All Year Table 10 Contents of a SAP BW base view with Visual result mapping (1) Appendices 189

201 12.3 USING THE ADVANCED CONDITIONS EDITOR The advanced conditions editor allows creating complex conditions in a purely graphical manner. This process is described below. On the left side of the screen we will find menus for creating various values that can appear as operands in the conditions: Constants. This menu allows creating constants of the various data types supported by Virtual DataPort. Functions. This menu allows creating an invocation to one of the functions permitted by Virtual DataPort. The functions can receive constants, attributes or the result of evaluating other functions as parameters. They return one result. The list of available functions and examples of use of each of them can be seen in the Syntax of Condition Functions section of the VQL Advanced Guide [VQL]. Attributes. This corresponds to the list of attributes of the view to which the selection is applied. The attributes can act as function parameters and/or appear directly as operands in the conditions. Figure 123 Editor of selection conditions On the right of the screen we will find menus to select the various operators that can appear in the conditions: Operators that can participate in the simple conditions. These are grouped according to the data type to which they can be applied. Logical operators (AND, OR, NOT). These are used to combine the different simple conditions in a Boolean expression. The center boxes of the screen allow constructing three types of elements from top to bottom: values that appear in the conditions, simple conditions and compound Boolean conditions. The box on the left of each group is a workspace Appendices 190

202 for creating new elements, while the box on the right shows the elements already created. The following subsections describe in more detail how each of these types of elements is created. Finally, the Selection condition box contains the selection condition created Creating Values for the Conditions To create a new constant value the following actions are required: 1. Select the data type of the constant in the Constants drop-down menu on the left side of the screen and click on it or drag&drop it to the work space where values are created (box on top left). 2. The type selected will appear in the workspace together with a text area to fill in the value of the constant. Where the constant belongs to a simple data type, the value required can be written directly in the text area. Where it belongs to a compound type (array or register), a new panel will be opened on clicking the button alongside the text area to edit the required constant values (see section ). If it belongs to the xml type, a panel will be opened on clicking the button that allows writing the required value required to be written or loading it from a file. 3. On clicking the > button, the new constant will appear in the list of values created (upper right-hand box). To create a new function-type value the following actions are required: 1. Select the required function in the Functions drop-down menu on the left side of the screen and click on it or drag&drop it to the work space for creating values (box on top left). 2. The function selected will appear in the workspace together with an area to fill in the value of each parameter of the function. The values of the parameters should be values present in the list of created values (box on top right). To assign a value already created as a parameter of a function drag&drop the value created to the parameter area. Press the > button that appears beside the function, and this will appear in the list of values created (box on top right). In our example, we will create a new constant value of the type Float. For this we will drag the type Float to the work space and fill in the value of same, in this case it will be 1000 (see Figure 124). Appendices 191

203 Figure 124 Creating a constant value Creating Simple Conditions To create a new simple condition the following actions are required: 1. Select the required simple condition operator in the drop-down menus on the right side of the screen and click on it or drag&drop it to the work space, where the simple conditions are created (left center box). 2. The operator selected will appear in the work space with an area to fill in its operands. The operands can be either attributes of the input view (present in the Fields drop-down menu of the left side of the screen) or values already created (the list of which is displayed in the box on the top right). To assign an attribute or a value already created as an operand of the condition drag&drop the element to the parameter area. Pressing the > button that appears beside the simple condition will cause this to appear in the list of simple conditions already created (centre right box). In our example, we will select the operator >= associated with the type Float and drag&drop same to the work space. We will use the attribute revenue as left operand. We will use the constant value Float(1000) created in the previous section as right operand. See Figure 125. Appendices 192

204 Figure 125 Creating a simple condition Creating Boolean Conditions To create a new Boolean condition the following actions are required: 1. Click on the required Boolean operator (AND, OR or NOT) in the drop-down menus on the right side of the screen. A Boolean condition will be created in the work space (left lower box). 2. The operands of the selected operator can be conditions previously created (right center box) and other Boolean conditions created beforehand. To assign a condition already created as an operand of the new Boolean condition drag&drop the condition to the operand area, press the > button that appears beside the Boolean condition, this then appears in the list of Boolean conditions already created (right lower box). 3. Drag&drop the condition to be added to the selection to the Selection Condition box. 4. Press ok to return to the selection view creation page, where the new condition will be displayed. In our example, we will not create compound Boolean conditions, so we will directly drag&drop from the simple condition created in the previous section to the Selection Condition box Creating Conditions that Use Compound-type Constants Denodo Virtual DataPort supports the modeling of data types with tree structure using register and array types (see Advanced VQL Guide [VQL]). It is possible to create constants of these compound types and to use them in the selection conditions. An explanation of how to do so will be given in this section through an example. Appendices 193

205 Figure 126 RevenueSum schema In the example, a base relation called RevenueSum will be used (see Figure 126). RevenueSum can be created as follows: 1. Using the Web Service datasource sales (see section 5.3.5), create a base view based on the getsumrevenuebytaxids operation. This operation has an input parameter which is an array of strings (the taxid of each client); and returns the sum of the revenue of all specified clients (float-type value). 2. The new base relation will be called RevenueSum. The name of its attributes will be changed to clients and totalrevenue, respectively. The clients attribute is an array of registers. More specifically, DataPort automatically generates an array type called getsumrevenuebyids_in0. Each element in this array will be a register, which is also automatically generated by DataPort and called getsumrevenuebyids_in0_string. The elements in this last type have a single texttype field and they will be renamed to taxid. Appendices 194

206 Figure 127 Compound values editor In this example, we will create a selection view using a condition on RevenueSum to obtain the total obtained by adding the revenues of the clients with taxids B and B The steps to follow are: 1. Select the option Combine Toolbox Projection/Selection. 2. Drag RevenueSum to the work area to create a new selection view. 3. Click the Set Condition option to open the selection conditions editor. The compound data type getsumrevenuebytaxids_in0 will now appear in the Constants dropdown on the left of the screen, along with the single data types. 4. Drag & drop the getsumrevenuebytaxids_in0 type to the value creation area and click the button. A new pop-up will appear as shown in Figure 127 to create a new getsumrevenuebytaxids_in0 type constant. 5. Click twice the + button beside Value. We need two elements of the getsumrevenuebytaxids_in0 type because the array must have two elements. 6. Click on the new elements to expand them (Figure 128) and edit the values of each array element (they will always be of a register type). Appendices 195

207 7. Enter the value B in the field taxid of the first register and B in the same field of the second register. Figure 128 Creating a value of type getsumrevenuebytaxids_in0 8. Click Ok to close the popup. The new compound value will appear in the creation area. 9. Click the > button to create the new constant (it is represented in VQL by {ROW(B ),ROW(B )}). See Conditions with Compound Values in the Advanced VQL Guide [VQL] for more information on ROW and {} constructors. The new constant can now be used to create the query condition required on the clients field. The creation process for this condition is the same as with single-type constants (see section ). Figure 129 shows the condition created in the Selection Condition box (represented in VQL by: clients={row(b ),row(b )}. Figure 130 shows the result of running the resulting selection view (see section to obtain information about how to execute views and navigate through the obtained results when these are of compound type). Appendices 196

208 Figure 129 Condition using compound-type values Figure 130 Result of executing the view RevenueSum Appendices 197

209 12.4 CONFIGURING LOGS Virtual DataPort uses the Jakarta-log4j [LOG4J] log system. This system allows log messages to be controlled with different levels of granularity and is configurable through configuration files. Some knowledge of log4j is required to modify the log configuration file. Briefly, Log4j allows defining a series of categories on which the log level will be specified independently. Log4j defines different log levels: DEBUG, INFO, WARN, ERROR and FATAL. NOTE: In production environments, we recommend using an ERROR log level. Lower levels degrade the system performance and are only recommended for debugging tasks. The Virtual DataPort Server logs are stored in the path DENODO_HOME/logs/vdp, where DENODO_HOME is the base installation path. The Administration tool logs are stored in the path DENODO_HOME/logs/vdp-admin. The log4j.xml configuration file of the Virtual DataPort Server logs can be found in the path DENODO_HOME/conf/vdp. Among other options, it is possible to modify the path where logs are stored and the log level of each of the categories defined in the application. See Log4J [LOG4J] documentation for more information. The log4j.xml configuration file of the Administration tool can be found in the path DENODO_HOME/conf/vdp-admin. Appendices 198

210 12.5 INSTALLING THE OPENAJAX MASHUP EDITOR Follow these steps to install the OpenAjax Mashup Editor. 1. Install an HTTP Server such as the Apache HTTP Server ( 2. Install PHP ( with the module Sqlite PDO and the ZIP extension. Open the $PHP_HOME/php.ini and uncomment or add the following lines : [PHP_PDO] extension=php_pdo.dll [PHP_PDO_SQLITE] extension=php_pdo_sqlite.dll [PHP_ZIP] extension=php_zip.dll 3. Download the following Subversion projects from these URLs: Or download them from the Web interface provided by this repository: / / / These URLs belong to the tag openajax.org of the gadgets, hub20 and repository subprojects of the OpenAjax Subversion repository. 4. Copy the downloaded projects to a folder of your HTTP server, i.e. $APACHE_HOME/htdocs/mashupapp. The Mashup Editor expects these directories to be located alongside: mashupapp/gadgets/ mashupapp/hub11/ mashupapp/repository/ Important: remove all the directories named.svn within these directories, because their content could interfere with the deployment of some widgets. 5. Rename the files mashupapp/gadgets/config/config.php.sample to config.php. and mashupapp/gadgets/config/subdomains.cfg.sample to subdomains.cfg 6. Edit the file mashupapp/gadgets/config.php: Change the value of SHINDIG_BASE_URL to Appendices 199

211 Change the value of REPOSITORY_URL to 7. Edit the file mashupapp/gadgets/config/subdomains.cfg: Replace every line cxx.openajax.org with the name of the server where the Mashup editor is deployed (i.e. if the host name is openajax.denodo.com, replace c1.openajax.org with openajax.denodo.com, c2.openajax.org with openajax.denodo.com, ) 8. Rename the file mashupapp/repository/config/config.php.sample to config.php 9. Edit the file mashupapp/repository/config.php: Change the value of REPOSITORY_URL to s IP address>/mashupapp/repository/ 10. Access to s IP address>/mashupapp/repository/install.php and click on the Start installation button to initialize the OpenAjax widgets database. If the last message is not Install finished! it means that the install process didn t finish correctly. Usually, the reason for this is that the request timed out. To solve this, change the value of the property max_execution_time to 60 in the php.ini file of your PHP installation and try again. Also remember that you should always use the same hostname in all the previous steps. Now the OpenAjax Mashup Editor is installed at s IP address>/www/gadgets/samples/newmashup.php and the Repository Browser, at <server s IP address>/mashupapp/repository/scaffold.php. Appendices 200

212 12.6 INSTALLING THE DENODO SOLUTION FOR MICROSOFT SHAREPOINT The Denodo Solution for Microsoft SharePoint is used by the Web Parts widgets published with DataPort (see section 7.6). It contains a DLL that, among other things, will connect to the auxiliary Web Service used by those widgets to obtain the data from a specific view. Important: you only need to deploy this solution the first time you want to deploy a Web Part. Follow these steps to deploy the solution: 1. Copy the file $DENODO_HOME/webapps/webpart-core/DenodoWebPart wsp to your SharePoint server. 2. Execute the following commands in your SharePoint server console: a. cd C:\Program Files\Common Files\Microsoft Shared\web server extensions\12\bin b. stsadm.exe -o addsolution filename "DenodoWebPart wsp" c. stsadm.exe -o deploysolution -name "DenodoWebPart wsp" -immediate -allcontenturls -allowcaspolicies The allowcaspolicies parameter is necessary because the solution contains Code Access Security (CAS) policies, to allow connections to an external server (auxiliary Web Service), write log information to files, read the XML configuration file of each Web Part, etc. d. stsadm.exe -o execadmsvcjobs you can omit this step if the Microsoft SharePoint Timer service is enabled. To list the existing solutions, execute: stsadm.exe -o enumsolutions To undeploy and delete the solution execute these commands in your SharePoint server console: 1. stsadm.exe -o retractsolution -name " DenodoWebPart wsp" -immediate -allcontenturls 2. stsadm.exe -o execadmsvcjobs you can omit this step if the Microsoft SharePoint Timer service is enabled. 3. stsadm.exe -o deletesolution -name " DenodoWebPart wsp" Appendices 201

213 12.7 TRANSFORMING INCOMING/OUTGOING SOAP MESSAGES WITH XSLT STYLESHEETS As explained in section 6.2.3, we can define XSLT stylesheets [XSLT] to transform incoming/outgoing SOAP messages. Thus, we avoid any modifications to existing SOAP clients. This section explains a possible way to generate these XSLT stylesheets more easily leveraging on some popular open-source tools. Before starting, we need to obtain some tools that will help us to define these transformations: SoapUI. ( Eclipse Web Tools. ( Set of Eclipse plugins that includes editors to create and test XSLT transformations. Specifically, we will use the XML Editor Tools and the XSL Developer Tools. First, we have to publish a new SOAP Web service from Virtual DataPort (section 6 explains how to do it). Make the structure of the new Web service as similar as possible to the existing Web service. By doing this, the XSLT stylesheets will be simpler: The operations of the new Web service should have the same parameters as the existing one. Change the names of operations and parameters so they match with the names of the existing Web service. You can also use FLAT / NEST operations (see section 5.4.5) to adjust the hierarchical structure of the published web service to thet of the target web service. For each operation, we need: For the Input XSLT Transformation : o A SOAP request sent by the existing Web service client. o A SOAP request expected by the new Web service. For the Output XSLT Transformation : o A SOAP response expected by the existing Web service client. o A SOAP response sent by the new Web service. To obtain these sample messages, we need the WSDL documents [WSDL] of the existing and the new Web services. If we have deployed the new Web service in the embedded Web container, we can obtain its WSDL at database name>/<name of service>/services/<name of service>?wsdl For both WSDL files, we have to obtain a sample SOAP request and a sample SOAP response. We can use SoapUI to do this: 1. Launch SoapUI. 2. Create a new project for each WSDL. In the New Project dialog, select the options Create Requests and Create MockService. After creating the project, the left-side panel contains a list of the operations of the Web service. 3. For each operation, double-click on Request to open the template of the request. 4. For each operation, double-click on Response to open the template of the response expected by the client. 5. Save each template: Right-click on each template s window and then, on Save as After this, we should have four SOAP files for each operation: a) A request sent by the existing Web service client. b) A response sent by the existing Web service. c) A request expected by the new Web service Appendices 202

214 d) A response sent by the new Web service. Now, for each operation, we have to generate two XSLT stylesheets: a) One XSLT stylesheet to transform the request sent by the existing client into what the new Web service expects. b) Another XSLT stylesheet to transform the response sent by the new Web service into what the existing client expects. There are many XML/XSLT editors to write XSLT transformations but in this appendix, we focus on XML Editor Tools and XSL Developer Tools from the Eclipse Web Tools. We can load the initial SOAP messages into the editor and create the XSLT stylesheets using some useful features of these editors: syntax highlighting and transformation execution/debugging. Once the XSLT transformations are finished, we have to execute them and validate their output against the schema defined in the WSDLs. There are two ways to validate the output of XSLT transformations: 1. Using Eclipse: extract the schema of the message from the WSDL and paste it in an.xsd file. Then, using the XML editor, execute the XSD validation of the transformed XML (just the content inside the <Body> tag) 2. Using SoapUI: in the appropriate window, replace the old XML with the result of the transformation.then, open the contextual menu and click on 'Validate'. Important note: if the schema of a Web service changes, their XSLT stylesheets have to be modified accordingly. Changes in the schema of a Web service include: a) Renaming existing operations. b) Removing operations. c) Adding / renaming / removing parameters of existing operations. In addition, if we add an operation to an existing Web service, its XSLT transformation will be empty. Appendices 203

215 12.8 JMS CONNECTION DETAILS: JNDI PROPERTIES AND CLIENT JARS This appendix provides information about how to connect Denodo with some of the most popular JMS servers: Configuration values required to configure a Denodo Web service or a JMS listener, so they can connect to a JMS server. List of the client jars required by the Denodo Web services and the JMS listeners to connect to the most popular JMS servers. The clients of JMS servers require these jars because the process of establishing the connection relies on proprietary classes of the vendor. Note: the first time a Web service is deployed into the embedded Web container of Virtual DataPort, copy these client jars into the $DENODO_HOME/resources/apache-tomcat/common/lib and restart the Virtual DataPort server (we only have to copy them once). If the Web service is deployed in an external application server, copy these jars into the /WEB-INF/lib directory of the war file, before deploying it. Note: the first time we create a JMS listener, we have to copy the client jars of the JMS vendor to the directory $DENODO_HOME/extensions/thirdparty/lib and restart the Virtual DataPort server Apache ActiveMQ The client jars for Apache ActiveMQ are located in $ACTIVE_MQ_HOME/lib: activemq-core.jar geronimo-j2ee-management.jar geronimo-jms_1.1_spec.jar The values for a default installation are: Property JNDI Provider URL Broker URL Property value tcp://acme:8161 tcp://acme:61616 Table 11 Default values for Apache Active MQ IBM WebSphere MQ The client jars for IBM WebSphere MQ are located in $WEBSPHERE_MQ_HOME/java/lib: com.ibm.mq.jar com.ibm.mqjms.jar com.ibm.mq.jmqi.jar dhbcore.jar jta.jar fscontext.jar providerutil.jar The properties for a default installation are: Appendices 204

216 Property Property value Host acme Port 1414 Channel SYSTEM.DEF.SVRCONN Queue Manager QM_acme Table 12 Default values for IBM WebSphere MQ OpenJMS The client jars for OpenJMS are located in $OPEN_JMS_HOME/lib: openjms-client.jar exolabcore.jar The JNDI properties for a default installation are: JNDI property name java.naming.provider.url java.naming.factory.initial transport.jms.connectionfactoryjnd IName Property value rmi://acme:1099 org.exolab.jms.jndi.initialcontextfact ory To subscribe to queues: JmsQueueConnectionFactory To subscribe to topics: JmsTopicConnectionFactory Table 13 Default JNDI properties for OpenJMS Note: The list of jars and JNDI properties are valid for OpenJMS They may change in future versions of OpenJMS Progress SonicMQ The client jars are located in $SONIC_MQ\Sonic\MQ8.0\lib: mfcontext.jar sonic_client.jar sonic_crypto.jar sonic_xa.jar sonic_xmessage.jar The JNDI properties for a default installation are: Appendices 205

217 JNDI property name java.naming.provider.url java.naming.factory.initial transport.jms.connectionfactoryjnd IName Property value acme:2506 com.sonicsw.jndi.mfcontext.mfcont extfactory ConnectionFactory Note: This connection factory name has to be defined in the JNDI service of SonicMQ. Table 14 Default JNDI properties for Progress SonicMQ Appendices 206

218 12.9 INSTALLING THE CONNECTORS FOR SAP ERP, SAP BW AND SAP BI To use BAPI (SAP ERP) data sources or Multidimensional data sources with SAP BW and SAP BI, we need to install their connectors in the system where Virtual DataPort runs Installing the Connector for SAP ERP (BAPI Data Sources) Follow these steps to install the SAP ERP connector: 1. Obtain the SAP Java Connector 3 (SAP JCo). It can be downloaded from section SAP Java Connector -> Tools & Services. If Virtual DataPort runs on a 32-bit O.S, download the 32-bit connector. If Virtual DataPort runs on a 32-bit JVM, on a 64-bit O.S, download the 32-bit connector. If Virtual DataPort runs on a 64-bit JVM, download the 64-bit connector. 2. Uncompress the downloaded package. 3. Copy the file sapjco3.jar to the directory $DENODO_HOME/extensions/thirdparty/lib 4. In Linux: copy the file libsapjco3.so to the directories $DENODO_HOME/jre/lib/i386 and $DENODO_HOME/jre/lib/i386/client. If Virtual DataPort is configured to use the JVM not included with the Denodo Platform, copy this file to the directories jre/lib/i386 and /jre/lib/i386/client of the JVM that runs Virtual DataPort. 5. In Windows: a. Copy the file sapjco3.dll to the directory $DENODO_HOME/extensions/thirdparty/lib b. Add the directory $DENODO_HOME/extensions/thirdparty/lib to the environment variable PATH. c. According to SAP note # ( if Virtual DataPort and SAP ERP are NOT running in the same host, we need to install the latest Visual Studio 2005 runtime libraries: i. Browse to ii. Scroll to section Affected Software. iii. Click on the link Microsoft Visual C Service Pack 1 Redistributable Package (KB973544) iv. Download one of the following platform-specific files depending on your scenario: 1. Virtual DataPort running on a 32-bit O.S: vcredist_x86.exe 2. Virtual DataPort running on a 32-bit JVM, on a 64-bit O.S: vcredist_x86.exe 3. Virtual DataPort running on a 64-bit JVM: vcredist_x64.exe 4. Itanium system: vcredist_ia64.exe v. Execute the downloaded file and follow the installation instructions. 6. To check that the JCo connector is working, execute java -jar "$DENODO_HOME/extensions/thirdparty/lib/sapjco3.jar" You should see something like Figure 131. Appendices 207

219 Figure 131 SAP Java Connector (JCo) test screen 7. Restart the Virtual DataPort server and the Administration Tool Installing the Connector for SAP BW and SAP BI (Multidimensional Sources) Follow these steps to install the SAP BW and SAP BI connector: 1. Obtain the SAP Business Intelligence Java Software Development Kit (BI Java SDK) distributed with SAP NetWeaver. 2. Uncompress the package and copy all its jars, including the jars in its subdirectories, to $DENODO_HOME/extensions/thirdparty/lib 3. Restart the Virtual DataPort server and the Administration Tool. Appendices 208