IBM Database Conversion Workbench 3.5

Transcription

1 3.5 IBM PureData System for Analytics to IBM dashdb Conversion Guide Version: 3.5 Last Updated: June 12th, 2015

2 Table of Contents 1. Introduction Prerequisites Overview of the Conversion Process Set Up Your DCW Environment Create a new DCW Conversion Project The IBM PureData System for Analytics to IBM dashdb Task Launcher DDL Extraction Extract DDL from a PureData System for Analytics Database Import the DDL into the DCW Project Compatibility Evaluation Format of the Compatibility Evaluation Report Generate the Compatibility Evaluation Report Prerequisites Evaluate Compatibility Support for large DDL files Submitting your Compatibility Evaluation Report to IBM Using the Compatibility Evaluation Report Prerequisites View through your web browser Overall Summary DDL Code Evaluation Unrecognized lines of code Support for BLU Acceleration Code Conversion Auto-convert Oracle syntax to dashdb compatible syntax Pre-requisites Run the auto-conversion Data Type Conversion Reviewing the converted code Deploy Objects to the dashdb Database Create a Connection to the dashdb database Executing the converted DDL on dashdb from DCW Pre-requisites Execute the DDL statements Execute the Converted DDL from the dashdb Console Appendix 1 - Data-Type Conversion Table Copyright IBM Corp All rights reserved Page 2 of 22

3 1. Introduction The IBM Database Conversion Workbench (DCW) integrates many of the tools used for converting from different database vendors to IBM database products, while providing an easy to use framework to take you through all the steps of the conversion process. DCW is provided as a plug-in to IBM Data Studio, a no-charge graphical tool based on an Eclipse platform. IBM Data Studio helps database developers and administrators manage, administer, and develop heterogeneous database environments for increased productivity and collaboration. For more information, visit: As an integrated workbench, DCW provides the following benefits to the user: - Common graphical interface with uniform look and feel for all phases of the conversion - A consolidated process with well-defined steps based on best practices - Faster enablement for client and partner teams in conversion methodology - Wizards that guide users through all steps of the conversion DCW offers the following step-by-step functions to facilitate the process. 1. DCW Task Launcher - An integrated help guide, which provides step by step instructions through the conversion process. 2. Compatibility Evaluation Provides a report of the estimated compatibility ratio of PureData System for Analytics SQL and NZPLSQL* statements with dashdb, outlining the major issues of the conversion, code that was auto-converted, as well as code that must be fixed manually. 3. Code Conversion - Auto-converts SQL statements to dashdb compatible syntax. This streamlines a time consuming process. * IMPORTANT NOTE: Support for analyzing NZPLSQL statements will be added to DCW in a future release.

4 2. Prerequisites The is a plug-in for IBM Data Studio. IBM Data Studio must first be installed on your machine before installing DCW. For the DCW 3.5 plug-in, the following requirements must be met: IBM Data Studio full client o Available for download at: For installation requirements and instructions, download the IBM DCW Plug-In Installation Guide: Copyright IBM Corp All rights reserved Page 4 of 22

5 3. Overview of the Conversion Process A standard conversion consists of the following steps: 1. Extract DDL from source database 2. Generate Compatibility Evaluation report from extracted DDL a. Estimate efforts based on incompatible syntax occurrences in the report 3. Automatically convert extracted DDL to syntax of the target database 4. Manually fix any additional incompatible syntax based on evaluation report 5. Create target database 6. Execute converted DDL to recreate database schema(s) 7. Extract data from source database 8. Deploy data into target database The Database Conversion Workbench provides an integrated environment with many functions to help take the user through the conversion steps described above. It is highly recommended for novice users to follow the process from beginning to end. While DCW supports the use of the various conversion functions separately, this approach is beyond the scope of this guide and recommended only for more advanced users. Copyright IBM Corp All rights reserved Page 5 of 22

6 4. Set Up Your DCW Environment 4.1. Create a new DCW Conversion Project In order to access the functions of the Database Conversion Workbench, you need to create a new DCW project. i. From Data Studio, navigate to menu File > New > Project then select the DCW Project wizard. ii. Enter the Project Name, select PureData System for Analytics as the Source Database and dashdb as the Target Database. iii. Click Finish. You might be asked to switch to the Database Conversion perspective. Click Yes to confirm the switch, and note that the following views are available: Project Explorer view lists all projects and relevant files. Data Source Explorer view lists all existing database connection profiles. Administration Explorer view lists all existing database connection profiles and provides access to database administration functions. DCW PureData System for Analytics to IBM dashdb Task Launcher launches the various wizards and help topics. Copyright IBM Corp All rights reserved Page 6 of 22

7 Copyright IBM Corp All rights reserved Page 7 of 22

8 5. The IBM PureData System for Analytics to IBM dashdb Task Launcher The Database Conversion Workbench comes with a built in Task Launcher, which acts as an assistant to launch the various steps of the conversion process. The Task Launcher is available after a DCW project has been created, and can be accessed at any time through the Help menu. Included with the task launcher are the DCW Help Topics which offer directions on how to access the various functions through your Project Explorer view, as well as helpful hints and step-by-step guidelines to follow for your conversion project. These help topics are automatically displayed when a process step is selected in the task launcher. Copyright IBM Corp All rights reserved Page 8 of 22

9 6. DDL Extraction 6.1. Extract DDL from a PureData System for Analytics Database You can use the nzdumpschema command or a third-party application to extract the DDL and save it to a file. After the DDL file is available, import it into your DCW project. The procedure below describes how to extract the DDL using the nzdumpschema command. Note: Refer to the PureData System for Analytics Knowledge Center for more information about the nzdumpschema command. i. From the command line in a PureData System for Analytics system, execute the command below. Note that the nzdumpschema command is located under under /nz/kit/bin/adm nzdumpschema -R <db_name> <output_file> where <db_name> is the database name and <output_file> is the name of the file into which the DDL is written. ii. Copy the <output_file> to the machine where the Database Conversion Workbench is installed Import the DDL into the DCW Project After you have extracted the DDL from your source database, import the file into your DCW project. i. In the Project Explorer view, right-click on your DCW Project and navigate to Database Conversion > Import a DDL file. The Import wizard opens. ii. Specify the directory of the DDL file and select the DDL file you want to import. Select the folder (DCW project) into which you want to import the DDL file. Click Finish. iii. The DDL file is imported into your DCW Project, and automatically converted to a.sql extension. Copyright IBM Corp All rights reserved Page 9 of 22

10 7. Compatibility Evaluation With a simple click of a button, the can analyze PureData System for Analytics DDL statements and create an assessment report which outlines the compatibility of the statements in use with the syntax of dashdb. The compatibility evaluation report includes an executive summary and a detailed list of DDL incompatibilities, as well as some suggested workarounds to fix the incompatible code. These issues can be addressed after the code conversion has been run in section Format of the Compatibility Evaluation Report By default, the report is saved in an encrypted format in a file with a.xmle extension. Section 7.3 explains how to decrypt and visualize the report Generate the Compatibility Evaluation Report The Compatibility Evaluation Report is generated by analyzing the DDL file of the source PureData System for Analytics database Prerequisites Ensure that you have a DDL file imported into your DCW project as outlined in section Evaluate Compatibility i. Right-click on the.sql file containing the DDL. Navigate to Database Conversion > Evaluate Compatibility. The Compatibility Evaluation wizard opens. Hint: You can run the compatibility evaluation on multiple DDL files at once. This generates a separate report for each DDL file. Note: DCW automatically splits DDL files larger than 5 MB into separate files for processing. This process generates a single merged report. See section Support for large DDL files for more information. Note: DCW has a 100 MB size restriction on the input DDL file. See section Support for large DDL files for more information. Copyright IBM Corp All rights reserved Page 10 of 22

11 ii. Select the Source SQL dialect as PureData System for Analytics and the Target SQL dialect as dashdb then click Finish. iii. The Compatibility Evaluation Report (.xmle file) is saved to your project folder Support for large DDL files For DDL files that are larger than 5 MB, DCW pre-processes the file by splitting the large DDL file into smaller more manageable components. DCW splits the large file into separate files, each of which is about 5 MB in size. The split files are created in a folder named <original_ddl_filename>_parts, which can be located within your DCW Project. To reduce the chance of error, DCW parses and processes each file separately and sequentially. Note: The progress of each file is displayed by DCW. Part 1 of 9 indicates that DCW split the original DDL file into 9 component files and is currently processing the 1 st file. Although processed separately, DCW generates a single and merged Compatibility Evaluation Report for the original large DDL file. At the end of the process, you receive a result message, which details any component files that were missed due to issues. You can rerun the Compatibility Evaluation on the missed files, which can be found in your DCW Project Submitting your Compatibility Evaluation Report to IBM The Compatibility Evaluation generates an encrypted.xmle report as <file_name>- <timestamp>_report.xmle. This encrypted report must be sent to IBM, and the decrypted report is ed back to you. Copyright IBM Corp All rights reserved Page 11 of 22

12 NOTE: The.xmle file does not contain any source code. It only contains the list of incompatibilities found and the number of occurrences in the original source code. Using your desired client, submit the.xmle file to NOTE: s sent by using gmail or hotmail accounts are not accepted. An response is sent back containing the HTML version of the Compatibility Evaluation Report, which can be viewed through your web browser Using the Compatibility Evaluation Report The Evaluation Report is a key part of the conversion process, and outlines the various compatibility issues in the syntax between PureData System for Analytics and dashdb. This section outlines how to use the Evaluation Report to correctly address any issues that are reported, after the code conversion has been run Prerequisites Ensure that you have the HTML version of the Evaluation Report. See section 7.2 Generate the Compatibility Evaluation Report for more details View through your web browser You can open the Evaluation Report using your preferred browser by double clicking on the file. You can also import the Evaluation Report into Data Studio, and use the native browser in Data Studio. i. Right-click on your DCW Project and select Import. The Import wizard opens. ii. Expand the General folder and select File System. Click Next. iii. Select Browse, and navigate to the directory in which the HTML Evaluation Report is located. After the folder is open, select the HTML Evaluation Report. Click Finish. iv. The report is now in your DCW Project. v. In your DCW Project, right-click on the HTML Evaluation Report and select Open With > Web Browser. The Evaluation Report opens Overall Summary This section of the report shows the compatibility percentage of all NZPLSQL* and DDL statements that can run natively on a dashdb database. NZPLSQL Statistics by Object Type shows a breakdown and count of objects that were detected, as well as the number of object types that require attention. Copyright IBM Corp All rights reserved Page 12 of 22

13 DDL Statistics by Statement Type shows the number of DDL statements that were detected, as well as the number of statements that require attention. * IMPORTANT NOTE: Support for analyzing NZPLSQL statements will be added to DCW in a future release. Total Number is the total count of statements and objects that was detected. The Number that Require Attention is the count of statements/objects that are not immediately transferrable to dashdb and require attention. Note: Statements and objects that can be auto-converted by DCW are not included in the count of Number that Require Attention DDL Code Evaluation This section outlines the DDL features that are not supported on dashdb. These statements must be altered or removed before execution on dashdb. This can be done after the code conversion has been run in section 8. You can expand or collapse each incompatible feature by clicking on the statement. For each issue identified, the report provides a list of all occurrences with the corresponding line number of the statement in the original DDL code snippet, the portion of the syntax that contains the error, and a recommended solution. Copyright IBM Corp All rights reserved Page 13 of 22

14 In addition to the features that are not supported, the report provides a DDL Features Informational section. These are statements that were automatically converted by DCW to the most appropriate dashdb compatible syntax, so no further actions are required. The list is provided to help consultants identify specific cases where the conversion applied might not be ideal Unrecognized lines of code The very last section of the report contains lines of code that the Compatibility Evaluation was not able to recognize. These statements might or might not be compatible with dashdb. You can check the code by referring to the line number that is listed Support for BLU Acceleration Since dashdb takes advantage of BLU Acceleration technology, the Compatibility Evaluation report contains two extra sections that list the BLU specific issues. The two BLU specific sections are: BLU Incompatibilities: Lists features that are not supported by column-organized tables. Look at the proposed solution in the report to learn how to fix it. BLU Stats: Contains the number of tables (CREATE TABLE statements) in the DDL file that can be created as column-organized tables. Note, in the Overall Summary section, the compatibility ratio does NOT account for BLU specific issues found. Copyright IBM Corp All rights reserved Page 14 of 22

15 8. Code Conversion The offers a powerful tool that can auto-convert well known Oracle syntax to dashdb compatible syntax Auto-convert Oracle syntax to dashdb compatible syntax Pre-requisites Ensure that you have a DDL file of the source database as outlined in section Run the auto-conversion i. Right-click on the DDL file of your source Oracle database in your DCW Project. Navigate to Database Conversion > Convert Code The Code Conversion wizard opens. ii. Select the Source SQL dialect and Target SQL dialect. Click Finish. iii. The converted DDL file is located in your DCW Project folder, named <originalfilename>- <timestamp>_converted.sql. Note: The code conversion automatically changes all statement terminators 8.2. Data Type Conversion As part of the automatic code conversion, PureData System for Analytics data types are converted to the most suitable data type in dashdb. Refer to Appendix 1 - Data-Type Conversion Table for a listing of all data type conversions Reviewing the converted code You can review the converted code in Data Studio by double clicking on the SQL file. Copyright IBM Corp All rights reserved Page 15 of 22

16 DCW does one of the following five things: i. Remove code fragment DCW comments out fragments of code that are not required on dashdb. These are tagged in the converted code with Code Fragment. ii. Auto-convert code DCW automatically makes updates to code to make it compatible with dashdb while maintaining the semantics of the original code. These statements are not tagged. iii. Requires Attention Code is marked as requires attention and is converted to dashdb compatible syntax so that the statement can compile and execute successfully. Although due to differences in behavior between dashdb and PureData System for Analytics, it might raise issues during runtime in very specific scenarios. For instance, DCW converts any NUMBER(N) where N is greater than 31 (max precision in dashdb) to NUMBER(31) which works for a great majority of scenarios. iv. Evaluation Issue DCW highlights statements that were identified in the Evaluation Report as being a potential issue. These are tagged in the converted code as DCW Evaluation Issue. Whenever possible, DCW modifies DDL statements in order to make them compatible with columnorganized tables. These cases are listed in the Compatibility Evaluation report as Informational only. At this point, you must manually edit the converted code to be fully compatible with dashdb. 1. Review code that requires attention. a. You can search the string requires attention by using CTRL+F. b. In the following example, the data type of col2 was truncated to VARBINARY(32672) which is the dashdb limit. In most cases, this is not a problem. However, DCW has identified it as requiring attention so that the user is aware that this change has been made, and might become an issue if this columns stores text values longer than the permitted limit. c. You can search the remaining converted code for the next statement that requires attention and review it appropriately. 2. Review code with Evaluation Issue a. You can search the string DCW Evaluation Issue by using CTRL+F. b. In the following example, the ALTER TABLE OWNER TO statement must be rewritten as a TRANSFER OWNERSHIP statement, as outlined in the Compatibility Evaluation Report. Copyright IBM Corp All rights reserved Page 16 of 22

17 c. You can search the remaining converted code for other statements with DCW Evaluation Issue and review them appropriately. Copyright IBM Corp All rights reserved Page 17 of 22

18 9. Deploy Objects to the dashdb Database This step involves executing the converted DDL statements against the target dashdb database Create a Connection to the dashdb database i. From the dashdb Console, retrieve the connection information from Connect > Connection Settings, on either the Connection without SSL page or the Connection with SSL page: Host name Port number Database name User ID Password ii. In the Database Conversion Workbench, go to the Administration Explorer tab and create a new connection. iii. Select DB2 for Linux, UNIX and Windows as the database manager, and fill out the connection information in the Properties section. Click Finish. Copyright IBM Corp All rights reserved Page 18 of 22

19 9.2. Executing the converted DDL on dashdb from DCW Pre-requisites Ensure that you have extracted the DDL statements from the source database. Ensure that the Convert DDL function has been executed on the original DDL file. If applicable, ensure that you reviewed and fixed any incompatibilities highlighted in the Compatibility Evaluation Report. Ensure that the target database has been created and it is set up properly. Ensure you have created a database connection as described on section Execute the DDL statements i. Using the Project Explorer view, navigate to your DCW project. ii. Double click on any SQL file to open the SQL Editor. 1. Use the Click here to connect to a database link at the top of the editor and select the target database. 2. Switch to the Validation tab at the bottom of the SQL Editor and change the statement terminator 3. Go back to the Editor tab and click on the green play button on the top right corner to run the DDL. 4. The results and status are displayed below the SQL Editor in the SQL Results tab. iii. Alternatively, you can use the DCW Run SQL Files command to execute multiple files at once. Copyright IBM Corp All rights reserved Page 19 of 22

20 1. Select the SQL file(s) that you want to run. You can also select the folder to run all of the SQL files in a folder. 2. Right-click and navigate to Database Conversion > Run SQL Files 3. Set your Statement delimiter Click Finish. 4. Select the Connection profile of your target database. Click Finish. 5. The scripts execute Execute the Converted DDL from the dashdb Console You also have the option to execute the converted DDL directly from the dashdb Copyright IBM Corp All rights reserved Page 20 of 22

21 Appendix 1 - Data-Type Conversion Table The following table summarizes the PureData System for Analytics data types that are supported on dashdb, as well as the data types that are automatically converted by DCW. Data types that are not supported must be manually converted to a dashdb supported type before data movement can be done. PureData System for Analytics Data Type IBM dashdb Data Type BYTEINT, INT1 Converted to SMALLINT BIGINT, INT8 BIGINT BOOLEAN, BOOL Converted to CHAR(1) CHAR(n), 0 <= n <= 254 CHAR(n) CHAR(n), 255 <= n <= Converted to VARCHAR(n) CHAR(n), n > Converted to VARCHAR(32762) DATE DATE DECIMAL, NUMERIC Converted to NUMERIC(18) DECIMAL(p, s), NUMERIC(p, s), 1 <= p <= 31 NUMERIC(p, s) DECIMAL(p, s), NUMERIC(p, s), p > 31 Converted to NUMERIC(31) DOUBLE PRECISION DOUBLE PRECISION FLOAT(p), 1<= p <= 6 REAL FLOAT(p), 7<= p <= 15 DOUBLE PRECISION INTEGER, INT4 INTEGER INTERVAL Not supported NCHAR(n), 1 <= n <= 63 NCHAR(n) NCHAR(n), 64 <= n <= 8168 Converted to NVARCHAR(n) NCHAR(n), n > 8168 Converted to NVARCHAR(8168) NVARCHAR(n), 1 <= n <= 8168 NVARCHAR(n) NVARCHAR(n), n > 8168 Converted to NVARCHAR(8168) REAL REAL SMALLINT (INT2) SMALLINT ST_GEOMETRIC Not supported TIME TIME TIME WITH TIME ZONE, TIMETZ Not supported TIMESTAMP TIMESTAMP VARBINARY(n), 1 <= n <= VARBINARY(n) VARBINARY(n), n > Converted to VARBINARY(32762) VARCHAR(n), 1 <= n <= VARCHAR(n) VARCHAR(n), n > Converted to VARCHAR(32672) Copyright IBM Corp All rights reserved Page 21 of 22

22 Copyright IBM Corporation 2015 All Rights Reserved. IBM Canada 8200 Warden Avenue Markham, ON L6G 1C7 Canada IBM, the IBM logo, ibm.com and Tivoli are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol ( or ), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at Copyright and trademark information at ibm.com/legal/copytrade.shtml Other company, product and service names may be trademarks or service marks of others. References in this publication to IBM products and services do not imply that IBM intends to make them available in all countries in which IBM operates. No part of this document may be reproduced or transmitted in any form without written permission from IBM Corporation. Product data has been reviewed for accuracy as of the date of initial publication. Product data is subject to change without notice. Any statements regarding IBM s future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only. THE INFORMATION PROVIDED IN THIS DOCUMENT IS DISTRIBUTED AS IS WITHOUT ANY WARRANTY, EITHER EXPRESS OR IMPLIED. IBM EXPRESSLY DISCLAIMS ANY WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE OR NON-INFRINGEMENT. IBM products are warranted according to the terms and conditions of the agreements (e.g. IBM Customer Agreement, Statement of Limited Warranty, International Program License Agreement, etc.) under which they are provided.