Supplied service Conversion service of an Oracle Warehouse Builder project to a corresponding Oracle Data Integrator project.

Transcription

1 White paper

2 INTRODUCTION OWB v.10.2 premier support ended on July OWB v.11.1 premier support will end on August OWB v.11.2 (available from September 2009) premier support will end on January No major enhancements are planned for OWB beyond the OWB 11.2 release. Earlier versions (9.2 and 10.1) of OWB will no longer be supported. In the latest Statement of Direction, Oracle announced following: No major enhancements are planned for Oracle Warehouse Builder beyond the OWB 11.2 release. OWB 11.2 continues to be available and supported by Oracle, and patches and bug fixes will continue to be offered at regular intervals. Oracle will continue to support OWB 11.2 for the full lifetime of Database 11g and the full lifetime of the next major database release ( Database 12 ) in accordance with Oracle s Lifetime Support Policies for Database releases. Future database releases beyond Database 12 would not be certified with OWB In other words, OWB will not be supported anymore beyond the release 12 of the Oracle database. Which means that you will have to migrate before Oracle Data Integrator (ODI) is now Oracle s Strategic Product for Data Integration. This means that customers will have to migrate from OWB to ODI, and needs to plan a manual or automated transfer of the OWB components to equivalent ODI components. Product overview OWB2ODI Converter acquires OWB repository s metadata and it generates corresponding ODI repository metadata. Algorithms in OWB are transferred in ODI, and OWB operators are transformed in corresponding ODI operators when present or are substituted by a custom solution. Supplied service Conversion service of an Oracle Warehouse Builder project to a corresponding Oracle Data Integrator project. Additional services Performance tuning, ODI, DBA consulting, Oracle GoldenGate and Life Cycle Management for ODI Supported product versions OWB v. 9.2 or higher ODI v. 10g or higher (up to v12c). Phase 1 - ASSESSMENT Page 2 of 20

3 Conversion process After the initial assessment phase with the client, a specific Converter s component extracts metadata from the OWB.mdl file. D&T uses the OWB2ODI Converter in its laboratories to automatically generate a corresponding ODI project from the original OWB project. The generated ODI project will be sent to the client for acceptance test and rollout to production. The conversion process consists of several tasks, which are divided into 5 phases: 1. Assessment 2. Conversion 3. Test 4. Parallel 5. Production PHASE 1: ASSESSMENT Phase 1 - ASSESSMENT Page 3 of 20

4 The assessment phase has following objectives: - To define the project framework, - To evaluate OWB project consistency (component number and type), - To define the conversion criteria. The Assessment phase is composed of the following tasks: - Task 1: conversion assessment and statistics report generation, - Task 2: handling exceptions, - Task 3: topology definition, - Task 4: conversion mode explanation, - Task 5: KMs definition, - Task 6: configuration management definition. Task 1 needs an OWB.mdl file export, which the client has to send to the D&T services centre. The.mdl file export must: - Contain all project Locations, - Contain the Configuration, if it is different from default, - Contain all mappings and all process flows in validated state, - Contain all dependences. After the generation of the report, the client and D&T organize a meeting to execute tasks 2 to 6. This meeting and relative decisions are a basilar rock upon which the entire conversion project will be built. Phase 1 - ASSESSMENT Page 4 of 20

5 Task 1 - Conversion assessment and statistics report generation The.mdl file supplied by the client has to be validated and must include location and configuration parameters. It represents the necessary input for converter analyzer. Converter Analyzer can perform a multitude of operations. It analyses whether any special operators exist, for which essential changes are required. For example, consider the case of the splitter operator that allows the flow to direct itself towards several targets in the same mapping. This operator has to be carefully managed because the Oracle Data Integrator does not allow a mapping flow to end in multiple targets. In this case, it generates as many interfaces as the Oracle Warehouse Builder operator targets. Then it deals with the analysis of each mapping. It starts from the target operator and ends on the leftmost operator, ensuring the entire mapping structure and its related operators remain unchanged. Phase 1 - ASSESSMENT Page 5 of 20

6 The Converter analyzer generates the following report on the OWB process flows and mappings: PROCESS FLOWS ACTIVITY_TYPE AMOUNT FIRST _FOLDER_NAME SECONDARY _FOLDER_NAME PF_NAME ACTIVITY _PER_PF PF_ STATE OR Folder_A Folder_1 Process flow A LOW TRANSFORMATION Folder_A Folder_1 Process flow A1.2 6 LOW MAPPING Folder_A Folder_1 Process flow A LOW SET_STATUS Folder_A Folder_1 Process flow A LOW SUBPROCESS Folder_A Folder_1 Process flow A LOW AND Folder_A Folder_1 Process flow A LOW ASSIGN 332 Folder_A Folder_1 Process flow A LOW FORK 252 Folder_A Folder_1 Process flow A LOW FILE_EXISTS 92 Folder_A Folder_1 Process flow A LOW USER_DEFINED 49 Folder_A Folder_1 Process flow A LOW WAIT 21 Folder_A Folder_1 Process flow A LOW END_LOOP 21 Folder_A Folder_1 Process flow A LOW FOR_LOOP 20 Folder_A Folder_1 Process flow A LOW ROUTE 14 Folder_A Folder_1 Process flow A LOW WHILE_LOOP 1 Folder_A Folder_1 Process flow A LOW Folder_A Folder_1 Process flow A LOW TOTAL Folder_A Folder_1 Process flow A LOW Folder_A Folder_1 Process flow A LOW Folder_A Folder_1 Process flow A LOW NUM_PROCESS_FLOWS Folder_A Folder_1 Process flow A MEDIUM Folder_A Folder_1 Process flow A MEDIUM MAX_NUM_ACTIVITY_IN_PF 200 Folder_A Folder_1 Process flow A LOW Folder_A Folder_1 Process flow A LOW NUM_PROCESS_FLOWS_LOW 720 Folder_A Folder_1 Process flow A LOW NUM_PROCESS_FLOWS_MEDIUM 180 Folder_A Folder_1 Process flow A LOW NUM_PROCESS_FLOWS_HIGH 100 etc. etc. etc. MAPPINGS OPERATOR_TYPE AMOUNT MODULE _NAME MAPPING _NAME OPERATOR _PER_MAPPING MAPPING _STATE TABLE Module 1 Mapping LOW EXPRESSION Module 1 Mapping LOW FILTER Module 1 Mapping LOW JOINER Module 1 Mapping LOW CONSTANT 687 Module 1 Mapping LOW INPUT_PARAMETER 630 Module 1 Mapping LOW AGGREGATOR 603 Module 2 Mapping LOW SET_OPERATION 447 Module 2 Mapping LOW SPLITTER 346 Module 2 Mapping MEDIUM PREMAPPING_PROCESS 323 Module 2 Mapping MEDIUM POSTMAPPING_PROCESS 183 Module 2 Mapping LOW SEQUENCE 151 Module 2 Mapping LOW FLAT_FILE 128 Module 2 Mapping LOW VIEW 89 Module 2 Mapping LOW DEDUPLICATOR 52 Module 2 Mapping LOW TRANSFORMATION 51 Module 3 Mapping MEDIUM SORTER 4 Module 3 Mapping MEDIUM Module 3 Mapping LOW TOTAL Module 3 Mapping LOW Module 3 Mapping LOW Module 3 Mapping LOW NUM_MAPPINGS Module 3 Mapping MEDIUM Module 3 Mapping LOW MAX_NUM_OPERATORS_IN_MAPPING 50 Module 4 Mapping LOW Module 4 Mapping LOW NUM_MAPPINGS_LOW 400 Module 4 Mapping LOW NUM_MAPPINGS_MEDIUM 300 Module 4 Mapping MEDIUM NUM_MAPPINGS_HIGH 300 etc. etc. Phase 1 - ASSESSMENT Page 6 of 20

7 D&T and the client discuss the conversion assessment & statistics report in a specific meeting organized to execute tasks 2 to 6. During the meeting, the client and D&T will compile the Roles and responsibilities matrix. This matrix assigns roles and responsibilities to the client and D&T for every task within the conversion process. Phase 1 - ASSESSMENT Page 7 of 20

8 Task 2 Exceptions handling This task is dedicated to deciding how to manage (work-around or manual conversion) any eventual case that cannot be automatically converted to ODI. A case can be represented by: A particular OWB operator in a mapping, A particular OWB activity or a particular transition condition in a process flow, An external workflow layer (not in OWB) used to execute the OWB project s components. These are extremely rare contingencies: most used OWB components are automatically converted. Task 3 Topology Definition In ODI, the definition of the topology, logical architecture, and physical architecture are necessary to indicate where the data are physically located. To correctly collect these informations, the OWB2ODI Converter analyzes the OWB Repository Metadata to obtain: Technology Machine where the data is located Database schema and/or file metadata ODI was developed to operate with any database technologies available (Oracle, DB2, Teradata, SQL Server, and many others). ODI provides the ability to refer to a database schema/user by its logical form. The logical user or logical schema is an abstract reference to a physical schema, which is defined within ODI as a real user in a specific database technology. Logical schemas and physical schemas are related through contexts. For example, the BUDGET logical schema may be associated with the BUDGET physical schema in the ORCL database through the CTX_BUDGET_ORCL context, etc. Everything is configured in ODI s topology section, which contains all the information needed to switch between the logical and physical side. The OWB2ODI Converter only works on logical schemas because the pointers to physical schemas are configured in the topology, and the context is assigned either at runtime or when conversions are executed. The result of that process shows how easy it is to generate a technology-independent code. As long as the right context is set up, everything will work properly. Please note: the logical schema is always associated with its own technology, although it can be easily moved to another by deleting it from the old platform and building it in the new one. Phase 1 - ASSESSMENT Page 8 of 20

9 Task 4 Conversion mode explanation This task is dedicated to give a detailed explanation of the conversion mode of every OWB component to the client s project team. The OWB2ODI Converter can convert: OWB mappings into ODI interfaces and ODI packages, OWB process flows logic into ODI native tools, ODI packages, ODI procedures and ODI load plans (available since ODI v ). Task 5 KMs definition ODI s capabilities to handle any RDBMSs are represented by its KMs (Knowledge modules). The OWB2ODI Converter has a specific console to set parameters and options related to the use of the KMs. The present task will affect ODI project s performances. The following is an example of how it is possible to customize the OWB2ODI Converter to choose the KMs better suited to the client s technology and needs. KMs customizable choosing grid DEFAULT SOURCE TARGET TYPE OF KNOWLEDGE MODULE OPERATING TECH TECH LOADING MODE Oracle SQLLDR ORACLE ORACLE INSERT SET BASED Oracle Incremental Update ORACLE ORACLE UPDATE SET BASED D&T Custom KM ORACLE ORACLE INSERT/UPDATE SET_BASED DSNTIAUL DB2 DB2 TRUNCATE/INSERT SET BASED Client Custom KM DB2 DB2 INSERT/UPDATE SET BASED FAIL OVER ROW BASED D&T customized KMs can also manage other aspects such as: Management of ANALYZE on target table, Management of the maximum number of allowable errors on target table, Management of loading HINT, Management of selecting HINT, Etc Phase 1 - ASSESSMENT Page 9 of 20

10 Task 6 Configuration management definition The last, but not the least important task. To obtain best results, it would be opportune to freeze any maintenance activity about OWB project to convert. The ideal scenario would be: - To rollout every OWB project undergoing modification into the client s production environment, - To align each of the client s environment (development, test and production), - To freeze any maintenance activity. The client has to decide if freezing is possible or not. If not, how does the client intend to execute configuration management of the OWB project to be converted. In this case, the client and D&T have to define a detailed operative protocol to be applied during the conversion period. During this task, the fixing protocol, which is the management process (roles, responsibilities, actions, etc.) needed to fix any bugs identified during the acceptance test task, has to be defined. Phase 1 - ASSESSMENT Page 10 of 20

11 PHASE 2: CONVERSION Phase 2 - CONVERSION Page 11 of 20

12 The Conversion phase has the following objectives: - To convert an original OWB project into a new ODI project, - To generate the new ODI project s metadata for the future import into the client ODI repository. The Conversion phase is composed of the following tasks: - Task 7: conversion of OWB Mapping and OWB Process Flow, - Task 8: formal test, - Task 9: ODI project metadata generation. Task 7 is the core of the entire conversion process and it is executed by the D&T services centre. Before starting the task, the following should be carried out: - To supply a.mdl file aligned with the latest OWB project version, - To supply a DB schema export aligned with the latest OWB project version, - To freeze any maintenance activity related to the OWB project to convert. Task 8 is executed by the D&T services centre and only concerns formal correctness of the new ODI project, because data are not available for a real test run. Task 9 is dedicated to the transmission of the new ODI project metadata generated by the Converter. Task 7 Conversion of OWB Mapping and OWB Process Flow According to task 3 Topology definition results, the D&T services centre carries out the ODI topology setting for its internal conversion environment. Then, all mappings are analyzed and a first transformation is applied whenever specific operators are found. This requires an onerous process in order to guarantee the proper functioning of the new Oracle Data Integrator flow, while keeping the semantic flow unchanged. This task, performed automatically by the OWB2ODI Converter, shows how costly and time-consuming it would be to carry out an entire manual conversion, not to mention the risk of introducing new mistakes due to haste or technical misunderstandings. Once additional mappings are normalized, recursive techniques are used to generate the operators tree of each mapping. The tree is retraced and each operator involved is transformed according to the new Oracle Data Integrator semantic. The correct topological information is maintained, considering the possible overruling of the location using a database of links or different schemas. The following tables indicate which OWB components the OWB2ODI Converter automatically converts and which components have to be converted via manual activity. Phase 2 - CONVERSION Page 12 of 20

13 From a conceptual point of view, OWB and ODI are similar, but they do not have equivalent features and have deep and significant differences. Thus, the OWB2ODI Converter does not handle some of the OWB components because they are rarely used or do not have a corresponding ODI function or because conversion would be too complex or ineffective or inefficient. OWB mapping: converted operators Aggregator Joiner Sorter Constant Mapping input/output parm Splitter Deduplicator (Distinct) Match-Merge Table Expression Materialized View Transformation External Table Pivot/Unpivot View Filter Pre/Post mapping process Anydata Cast (since 11.1) Flat File (File multirecord) Key Lookup Sequence Set Operation OWB mapping: not handled operators Dimension Expand object Pluggable mapping Cube Varray iterator Queue (11.2) Construct Name and address Subquery filter (11.2) Data generator Table function LCR cast/splitter (11.2) OWB process flow: converted activities Data auditor FTP And / Or Mapping Manual End Subprocess Notification End Loop Transform Set status For Loop Assign Sqlplus While Loop User Defined Fork File Exists Wait Route Phase 2 - CONVERSION Page 13 of 20

14 OWB process flow: converted transition conditions Success Error Complex OWB process flow: not handled activities Web services (11.2) EJB / Java class (11.2) OMB plus (11.2) OWB process flow: not handled transition conditions Warning* Extended Since ODI does not have a warning status, the OWB warning transition condition is handled depending on each client s specific needs. This is a possible topic to be covered in the assessment meeting. The OWB Fork activity is converted by using the ODI Load plan feature that is only available from ODI v onwards. Therefore, in order to manage and convert all of the most used and useful activities in the Process Flow, we need an ODI version that is equal or greater than ODI v Task 8 Formal test This is the first formal non-regression test executed on empty data structures, without data. This is carried out by the D&T services centre. Task 9 ODI project metadata generation The last task of the conversion process is dedicated to the ODI project metadata generation. These metadata are included in the.xml files that are delivered to the client. Phase 2 - CONVERSION Page 14 of 20

15 PHASE 3: TEST The test phase has following objectives: - To compare new ODI project results with old OWB project results (acceptance test), - To tune new ODI project performances. Phase 3 - TEST Page 15 of 20

16 The test phase is composed of the following tasks: - Task 10: test environment arrangement, - Task 11: acceptance test, - Task 12: performance tuning. Task 10 Test environment arrangement During this task, each test environment s element has to be arranged. In detail it is necessary: - To install OWB product, - To install ODI product, - To copy OWB project s components, - To import the two.xml files supplied from D&T services centre into ODI repository, - To copy the production DB twice: one copy for OWB project and another copy for ODI project. Task 11 Acceptance test The ODI project is tested to ensure that regression issues are not present. In detail, a complete parallel run of the two projects is executed and then respective results are compared to verify their matching. Each incorrect result is investigated to determine relative causes. The fixing protocol established during the client/d&t initial meeting (Phase 1 Assessment) is applied for each bug detected. Task 12 Performance tuning The last step of the Test phase is related to performances. During this task, the ODI Project s performances are tuned with data base administrator support. Phase 3 - TEST Page 16 of 20

17 PHASE 4: PARALLEL The parallel phase has following objectives: - To assure new ODI project correctness in Production environment such as the Test environment, - To compare and eventually tune new ODI project performances compared to the original OWB project in Production environment. Phase 4 - PARALLEL Page 17 of 20

18 The test phase is composed of the following tasks: - Task 13: parallel environment arrangement, - Task 14: verification test, - Task 15: performance verification. Task 13 Parallel environment arrangement During this task, each parallel environment s element has to be prepared in order to: - Deploy ODI project, - Copy production DB. Obviously, the DB copy has to be executed just before the parallel run is started. Task 14 Verification test Theoretically, this is a redundant and unnecessary task, but experience teaches us that a parallel production period must be carried out in order to locate any possible problems that may affect the production environment (configuration, privileges, missing patches, etc.). The comparison of results of two projects (just executed during the Test phase) is executed again. Task 15 Performance verification Theoretically, this is also a redundant and unnecessary task, but experience teaches us that a parallel comparison must be carried out between the original OWB project and new ODI project performances in the production environment. Phase 4 - PARALLEL Page 18 of 20

19 PHASE 5: PRODUCTION The production phase has following objectives: - To remove the original OWB project and its pertinences (OWB installation, DB schema, etc.) from production environment, - To switch the scheduling tool from the original OWB project to the new ODI project. The test phase is composed of the following tasks: - Task 16: production environment cleaning, - Task 17: switching from OWB to ODI. Task 16 Production environment cleaning The cleaning consists of: - OWB uninstallation, - DB schema deletion, - Removal of any other software components that are not necessary to the new ODI project running. Task 17 Switching from OWB to ODI This consists of physical switching from the old OWB project to the new ODI project. Phase 5 - PRODUCTION Page 19 of 20

20 CONTACTS For more information about OWB2ODI Converter, please visit or visit or send an to or call to speak to a D&T representative. Database & Technology s.r.l. Largo Promessi Sposi, Milano, Italy CONTACTS Page 20 of 20