Oracle Data Integrator: Administration and Development Volume I Student Guide

Transcription

1 Oracle Data Integrator: Administration and Development Volume I Student Guide D48459GC30 Edition 3.0 December 2007 D53463

2 Authors Laura Hofman Miquel FX Nicolas Technical Contributor and Reviewer Sharath Bhujani Editors Vijayalakshmi Narasimhan Daniel Milne Amitha Narayan Publishers Veena Narasimhan Sujatha Nagendra Nita Brozowski Copyright 2007, Oracle. All rights reserved. Disclaimer This document contains proprietary information and is protected by copyright and other intellectual property laws. You may copy and print this document solely for your own use in an Oracle training course. The document may not be modified or altered in any way. Except where your use constitutes "fair use" under copyright law, you may not use, share, download, upload, copy, print, display, perform, reproduce, publish, license, post, transmit, or distribute this document in whole or in part without the express authorization of Oracle. The information contained in this document is subject to change without notice. If you find any problems in the document, please report them in writing to: Oracle University, 500 Oracle Parkway, Redwood Shores, California USA. This document is not warranted to be error-free. Restricted Rights Notice If this documentation is delivered to the United States Government or anyone using the documentation on behalf of the United States Government, the following notice is applicable: U.S. GOVERNMENT RIGHTS The U.S. Government s rights to use, modify, reproduce, release, perform, display, or disclose these training materials are restricted by the terms of the applicable Oracle license agreement and/or the applicable U.S. Government contract. Trademark Notice Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners.

3 Contents 1 Oracle Data Integrator Architecture Objectives 1-2 Architecture: Overview 1-3 What Is Oracle Data Integrator? 1-4 The Oracle Data Integrator Architecture 1-5 Components 1-7 Graphical Modules 1-8 Run-Time Components 1-9 Metadata Navigator 1-10 Components: A Global View 1-11 Oracle Data Integrator Repositories 1-12 Master and Work Repositories 1-13 Repository Setup: Example 1-15 Summary Topology Concepts Objectives 2-2 Topology: Overview 2-3 What Is Topology? 2-4 What Is in the Topology? 2-5 Data Servers and Physical Schemas 2-6 What Is a Data Server? 2-7 Important Note 2-9 What Is a Physical Schema? 2-10 Physical Schemas: Properties 2-11 Concepts in Reality 2-12 Important Note 2-13 Infrastructure (Production): Example 2-15 Physical Architecture in Oracle Data Integrator 2-16 Infrastructure (Development): Example 2-17 The Fragmentation Problem 2-18 Important Note 2-19 The Infrastructure Example Again 2-20 Defining the Physical Architecture 2-21 The Infrastructure Example Again 2-22 Physical Architecture in Oracle Data Integrator 2-23 Logical Schemas and Contexts 2-24 What Is a Logical Schema? 2-25 iii

4 Important Note 2-27 Rule #4 Explained 2-28 Information System Example Again 2-29 Defining the Logical Architecture 2-30 Logical Versus Physical Architecture 2-31 Note: Design Time Versus Run Time 2-32 What Is a Context? 2-33 Important Note 2-35 Defining Contexts 2-36 Mapping Logical and Physical Resources 2-37 Note 2-38 Agents 2-39 What Is an Agent? 2-40 Agents 2-41 Important Note 2-42 Infrastructure with Agents: Example 2-43 Defining Agents 2-44 Defining a Topology 2-45 Planning the Topology 2-46 Matrix of Logical/Physical Mappings 2-47 Summary Connecting to Your Data Objectives 3-2 Topology Manager 3-3 What Is Topology? 3-4 What Topology Manager Contains 3-5 Topology Manager: Overview 3-6 The Physical Architecture View 3-8 Data Servers and Physical Schemas 3-9 Prerequisites for Connecting to a Server 3-10 Important Note 3-11 Creating a Data Server 3-12 Creating a Data Server: JDBC 3-13 Creating a Data Server: JNDI 3-14 JDBC Driver 3-15 The JDBC URL 3-16 Drivers and URLs: Examples 3-17 Testing a Data Server Connection 3-18 Note: Testing the Connection 3-19 Creating a Physical Schema 3-20 Note: Dedicated Area for Work Schema 3-22 Physical Agents 3-23 iv

5 Creating a Physical Agent 3-24 Important Note 3-25 Summary Defining the Logical Architecture Objectives 4-2 Note: Planning Ahead 4-3 Topology Manager 4-4 Topology Manager: Overview 4-5 Logical Architecture and Context Views 4-6 Defining Contexts 4-7 Defining a Context 4-8 Declaring Logical Schemas and Agents 4-9 Creating a Logical Schema 4-10 Creating a Logical Agent 4-11 Linking the Logical and Physical Architecture 4-12 Note: Unresolved Agents and Schemas 4-13 Summary Setting Up a New Project Objectives 5-2 Projects 5-3 What Is a Project? 5-4 Oracle Data Integrator Projects: Overview 5-5 What Does a Project Represent? 5-6 Creating a New Project 5-7 Folders 5-8 What Is a Folder? 5-9 Creating a New Folder 5-10 Organizing Projects and Folders 5-11 Knowledge Modules 5-12 What Is a Knowledge Module? 5-13 Knowledge Module: Types 5-14 Which Knowledge Modules Do You Need? 5-15 Identifying Knowledge Modules 5-16 IKM and LKM Strategies and Methods 5-17 Knowledge Modules: Examples 5-18 Importing a New Knowledge Module 5-19 Replacing Existing KMs 5-20 Replacing an Existing KM 5-21 Markers 5-22 What Is a Marker? 5-23 Tagging Objects with Markers 5-24 v

6 Removing Markers 5-25 Changing an Object s Markers 5-26 Marker Groups 5-27 Project and Global Markers 5-28 Creating a Marker Group 5-29 Marker Options 5-30 Exchanging Objects 5-31 Exporting and Importing 5-32 Exporting an Object 5-33 Importing an Object 5-34 ID Numbers: Overview 5-35 Import Types 5-36 Choosing the Import Mode 5-38 Summary Releasing Projects Objectives 6-2 Scenarios 6-3 What Is a Scenario? 6-4 Properties of Scenarios 6-5 Managing Scenarios 6-6 Scenario-Related Tasks 6-7 Generating a Scenario 6-8 Regenerating a Scenario 6-9 Generation Versus Regeneration 6-10 Executing a Scenario from the GUI 6-11 Executing a Scenario from a Command Line 6-12 Executing a Scenario from a Project 6-13 Exporting a Scenario 6-14 Preparing for Deployment 6-15 Preparing Scenarios for Deployment 6-16 Automating Scenario Management 6-17 Summary Oracle Data Integrator Model Concepts Objectives 7-2 Models 7-3 Relational Paradigm 7-4 What Is a Model? 7-5 The Relational Paradigm 7-6 Relational Model: Tables and Columns 7-7 Relational Model: Keys 7-8 Relational Model: Foreign Keys 7-9 vi

7 Relational Model: Constraints 7-10 Relational Model: Indexes 7-12 Relational Model Support in Oracle Data Integrator 7-13 Additional Metadata in Oracle Data Integrator 7-14 Reverse Engineering 7-16 What Is Reverse Engineering? 7-17 Methods for DBMS Reverse Engineering 7-18 Other Methods for Reverse Engineering 7-19 Standard Versus Customized Reverse Engineering 7-20 Note 7-21 Fleshing Out Models 7-22 Elements Missing from Models 7-23 Fleshing Out Models 7-24 Summary Creating and Reverse Engineering Models Objectives 8-2 Creating Models 8-3 What Is a Model? 8-4 What Is Reverse Engineering? 8-5 Methods for DBMS Reverse Engineering 8-6 How to Create a Model by Reverse Engineering Creating a New Model 8-8 Creating and Naming a New Model 8-9 Note Setting Up a Reverse Engineering Strategy 8-11 How to Define a Reverse Engineering Strategy Starting the Reverse Engineering Process 8-14 How to Start the Reverse Engineering Process 8-15 Selective Reverse Engineering 8-16 Summary Fleshing Out Data Models Objectives 9-2 Organizing Models 9-3 What Is a Model Folder? 9-4 Creating a Model Folder 9-5 What Is a Submodel? 9-6 Creating a Submodel 9-7 Organizing Data Stores into Submodels 9-8 Setting Up Automatic Distribution 9-9 Creating Data Stores 9-10 Creating a Data Store in a Model 9-12 vii

8 Adding Columns to a Data Store 9-13 Constraints 9-14 Constraints in Oracle Data Integrator 9-15 Creating Mandatory Columns 9-16 Creating a Mandatory Column 9-17 Creating Keys 9-18 Creating a Key 9-19 Checking a Key 9-20 Note: Synchronous Check Limitations 9-21 Creating References 9-22 Creating a Reference 9-23 Creating a Simple Reference 9-24 Creating a Complex Reference 9-25 Checking a Reference 9-26 Creating Conditions 9-27 Creating a Condition 9-28 Checking a Condition 9-29 Summary Enforcing Data Quality with Oracle Data Integrator Objectives 10-2 Data Quality 10-3 Why Data Quality? 10-4 When to Enforce Data Quality 10-5 Data Quality in Source Applications 10-6 Data Quality Control in the Integration Process 10-7 Data Quality in the Target Applications 10-8 Business Rules for Data Quality 10-9 Data Quality Business Rules From Business Rules to Constraints Enforcing Data Quality What Is Data Quality Control? Data Quality System: Overview Static and Flow Controls: Differences Data Quality Control: Properties Synchronous Control Summary Exploring and Auditing Your Data Objectives 11-2 Overview 11-3 When and Why? 11-4 An Overview of the Process 11-5 viii

9 Exploring Your Data 11-6 Displaying the Contents of a Data Store 11-7 Viewing the Distribution of Values 11-8 Analyzing the Contents of a Data Store 11-9 Constructing Business Rules Defining Business Rules in Oracle Data Integrator From Business Rules to Constraints Deducing Constraints from Data Analysis Testing a Constraint Auditing Data Quality Auditing a Model or Data Store How to Review Erroneous Records Using a Package Summary Building an Active Integration Hub Objectives 12-2 The Active Integration Hub 12-3 What Is an AIH? 12-4 The Active Integration Hub 12-5 How to Create an AIH Designing an AIH Structure 12-7 What Is a Diagram? 12-8 Creating and Naming the New Diagram 12-9 How to Design the Diagram Important Note How to Make the Diagram More Readable Implementing the AIH Generating DDL Scripts Important Note Merging Data and Events into the AIH Interface Generation: Overview Important Note Propagating Data and Events from the AIH Generating Interfaces OUT: Overview Important Note Going Further Summary Interface Design: One-to-One Interfaces Lesson Objectives 13-2 Interfaces: An Overview 13-3 What Is an Interface? 13-4 ix

10 Creating a One-to-One Interface Creating and Naming the Interface 13-6 Creating and Naming an Interface 13-7 Important Note Defining the Target Data Store 13-9 Defining the Target Data Store Important Note Defining the Source Data Store Defining the Source Data Store Important Note Defining the Mappings What Is a Mapping? Defining the Mappings Valid Mapping Types Important Note Saving the Interface Saving the Interface Important Note Summary Interface Design Objectives 14-2 Multiple Sources and Joins 14-3 What Is a Join? 14-4 Multiple Source Data Stores 14-5 Important Note 14-6 Creating a Join Manually 14-7 Advanced Joins 14-8 Types of Joins 14-9 Setting Up a Join Filtering Data What Is a Filter? Filters in Oracle Data Integrator Defining a Filter Manually Setting Up a Filter Summary Executing and Debugging Interfaces Objectives 15-2 Interface Execution 15-3 How to Execute an Interface 15-4 What Happens at Run Time 15-5 Monitoring Interfaces 15-7 x

11 Operator: Viewing the Log 15-8 Using Operator 15-9 Sessions, Steps, Tasks: The Hierarchy Viewing Sessions and Tasks Important Note How to Monitor Execution of an Interface Troubleshooting a Session How to Troubleshoot a Session Identifying the Error Reviewing the Code How to Fix the Code and Restart the Session Fixing the Interface Keys to Reviewing the Generated Code Common Errors and Symptoms Important Note Preventing Errors Tips for Preventing Errors Summary Interface Design Objectives 16-2 Essential Concepts in Data Quality 16-3 What Is Data Quality Control? 16-4 What Is a Constraint? 16-5 Defining and Enforcing Data Quality 16-6 Data Quality System: Overview 16-7 What Can be Checked? 16-8 Data Quality Control: Properties 16-9 Static/Flow Control: Differences How to Enforce Data Quality in an Interface Enabling Static/Flow Control for an Interface Setting Up Static/Flow Control How to Enable Static/Flow Control Setting the Options How to Set the Options Selecting Which Constraints to Enforce How to Select Which Constraints to Enforce How to Select Which Constraints to Check Differences Between Control Types Reviewing Erroneous Records How to Review Erroneous Records Note Summary xi

12 17 Defining Simple Procedures Objectives 17-2 Procedures: Overview 17-3 What Is a Procedure? 17-4 Procedure: Examples 17-5 Using Procedures: Overview Creating a Procedure 17-8 How to Create a New Procedure Adding Commands Creating a Command Arranging Steps in Order Which Parameters Should Be Set? Valid Types of Commands More Elements Why Use a Source Command? Adding Options Types of Options How to Create a New Option How to Make a Command Optional Using an Option Value in a Command Running a Procedure Procedure Execution Using Operator to View Results Note: Procedure Steps Are References Summary Package Design 1 Objectives 18-2 Packages: Overview 18-3 What Is a Package? 18-4 How to Create a Package Creating and Naming a Package 18-6 How to Create and Name a Package 18-7 The Package Diagram 18-8 Package Diagram Toolbar Adding Steps to the Package Steps How to Create an Interface Step Note: Interfaces Are Reusable Note: All Steps from Same Project What Is an Oracle Data Integrator Tool? How to Create an ODI Tool Step xii

13 Note: Tool Steps Are Not Reusable Note: Other Step Types Exist Arranging Package Steps in a Sequence Sequencing Steps A Simple Package Note: One Starting Step How to Sequence Package Steps Note: Error Button Executing a Package Note: Unit Testing Summary Open Tools Overview 19-2 Objectives 19-3 What Are Open Tools? 19-4 Installing Open Tools 19-6 Installing Open Tools 19-7 Adding a New Open Tool 19-9 Removing an Open Tool Using Open Tools Using Open Tools in a Package Using Open Tools in a Procedure or in a KM Developing Your Own Open Tools Open Tools at Run Time Reference Information Summary Variables and Sequences Objectives 20-2 Variables 20-3 What Is a Variable? 20-4 Variable Scope 20-5 Note: Variables Across Projects 20-6 Note: Variables at Run-Time 20-7 Value Persistence: Action Types 20-8 Configuring a Variable # Refreshing a Variable with SQL Configuring a Variable # How to Define a Variable Note: Case-Sensitivity Using Variables Using a Variable in Code xiii

14 Binding Versus Substitution How to Create a Variable Step Variable Steps Sequences What Is a Sequence? Properties of Sequences How to Create a Sequence Referring to Sequences Note: Sequences Updated by Agent Using Sequences in Mappings Correctly Using Oracle Data Integrator Sequences in Mappings Populating Native Identity Columns Note Summary User Functions Objectives 21-2 Overview 21-3 Simple Example # Simple Example # What Is a User Function? 21-6 Creating User Functions 21-7 Properties of User Functions 21-8 How to Create a User Function 21-9 Defining an Implementation Syntax and Implementations Using User Functions User Functions at Design Time User Functions at Run Time Note: Functions in Execution Log Summary Package Design 2 Objectives 22-2 Types of Package Steps 22-3 Basic Step Types 22-4 Advanced Step Types 22-5 Procedure Steps 22-6 How to Create a Procedure Step 22-7 Model, Submodel, and Data Store Steps 22-8 How to Create Model, Submodel, and Data Store Steps 22-9 Models, Submodels, and Data Store Steps Note: Take Care with Model Steps xiv

15 Variable Steps How to Create a Variable Step Variable Steps Controlling the Execution Path Controlling Execution Error Handling How to Create a Loop The Advanced Tab Summary Interface Concepts Objectives 23-2 Interfaces 23-3 What Is an Interface? 23-4 Business Rules for Interfaces 23-5 What Is a Mapping? 23-6 What Is a Join? 23-7 What Is a Filter? 23-8 What Is a Constraint? 23-9 Where Are the Rules Defined? Behind the Rules How Is Oracle Data Integrator Different? A Business Problem Implementing the Rules Integration Process Process Details Process Implementation # Process Implementation # Process Implementation # What Is the Staging Area? Execution Location Note From Business Rules to Processes Knowledge Modules What Is a Knowledge Module? Code Generation KM Types Used in Interfaces Important Note Summary Interface Design Objectives 24-2 An Overview of the Flow 24-3 xv

16 What Is the Flow? 24-4 Note 24-5 What Defines the Flow? 24-6 An Example of Flow 24-7 The Scenario 24-8 The Basic Process Selecting a Staging Area What Is the Staging Area? Case Study: Placing the Staging Area Note: Staging Area Must Be an RDBMS How to Specify the Staging Area Case Study: Flow # Case Study: Flow #1 in Oracle Data Integrator Case Study: Flow # Case Study: Flow #2 in Oracle Data Integrator Case Study: Flow # Case Study: Flow #3 in Oracle Data Integrator Note: Staging Area Syntax Configuring Filters, Joins, and Mappings Options for Filters, Joins, and Mappings Setting Options for Filters, Joins, and Mappings How to Disable a Transformation Note: Disabling Transformations How to Enable a Mapping for Inserts or Updates Note: Update Keys for Flow Control What Is an Update Key? How to Define the Update Key How to Change the Execution Location Why Change the Execution Location? Execution Location and Syntax Note: Moving the Staging Area Selecting the KM Which KMs for Which Flow? More on KMs Identifying IKMs and LKMs Flow # Flow # Flow # How to Specify an LKM Note: Default KMs How to Specify an IKM Common KM Options xvi

17 Note: The Staging Area Trade-off Summary Interface Design: Advanced Business Rules Objectives 25-2 Overview 25-3 Business Rules in Interfaces 25-4 Business Rule Elements 25-5 More Elements 25-6 Using Variables 25-7 Using a Variable in Code 25-8 Binding Versus Substitution Note: Case-Sensitivity Using Sequences Referring to Sequences Note: Sequences Updated by Agent Using Sequences in Mappings Correctly Using Oracle Data Integrator Sequences in Mappings Populating Native Identity Columns Note Using User Functions Using Substitution Methods Substitution Methods: Examples The Expression Editor Note Summary Monitoring Sessions Objectives 26-2 Operator: Overview 26-3 Overview 26-4 Operator: Viewing the Log 26-5 Sessions 26-6 What Is a Session? 26-7 Sessions, Step, Tasks: The Hierarchy 26-8 Viewing Sessions 26-9 Viewing Steps Viewing Tasks Viewing Tasks: Execution Tab Errors, Inserts, Updates, Deletes Managing Sessions What You Can Do with Sessions Monitoring a Session xvii

18 Restarting a Session Note: Transactions Affect Tasks Important Note Stopping a Session Important Note Purging the Session Log Troubleshooting a Session How to Troubleshoot a Session Finding the Error Reviewing the Code Note: Use Squirrel Note: Java Exceptions Fixing the Code and Restarting the Session Fixing the Interface Summary SOA Integration with Oracle Data Integrator Overview 27-2 Objectives 27-3 Web Services In Action 27-4 Data Services 27-5 What Are Data Services? 27-6 Generation of Data Services (I) 27-7 Generation of Data Services (II) 27-8 Data Services In Action 27-9 Setting Up Data Services (I) Setting Up Data Services (II) Setting Up Data Sources in a Web Services Container (I) Setting Up Data Sources in a Web Services Container (II) Setting Up Data Sources in a Web Services Container (III) Configuring the Topology (I) Configuring the Topology (II) Configuring the Topology (III) Configuring the Topology (IV) Configuring the Topology (V) Configuring the Topology (VI) Setting Up the Model (I) Setting Up the Model (II) Setting Up the Model (III) Setting Up the Model (IV) Setting Up the Model (V) Setting Up the Model (VI) Setting Up the Model (VII) xviii

19 4. Generating, Deploying, and Testing Data Services (I) Generating, Deploying, and Testing Data Services (II) Generating, Deploying, and Testing Data Services (III) Generated Services (I): Overview Generated Services (II): Overview Generated Services (III): Overview Generated Services (IV): Overview Generated Services (V): Overview Testing Data Services (I) Testing Data Services (II) Testing Data Services (III) Testing Data Services (IV) Public Web Services What Are Public Web Services? (I) What Are Public Web Services? (II) Public Web Services in Action Installing Public Web Services (I) Installing Public Web Services (II) Using Public Web Services (I) A Simple SOAP Request for the OdiInvoke Web Service (I) A Simple SOAP Request for the OdiInvoke Web Service (II) A Simple SOAP Request for the OdiInvoke Web Service (III) A Simple SOAP Request for the OdiInvoke Web Service (IV) A Simple SOAP Request for the OdiInvoke Web Service (V) A Simple SOAP Request for the OdiInvoke Web Service (VI) A Simple SOAP Response for the OdiInvoke Web Service Invoking Web Services OdiInvokeWebService Tool CountryInfoService Web Service: Example Invoking a Web Service (I) Invoking a Web Service (II) Invoking a Web Service (III) Invoking a Web Service (IV) Processing a Web Service Response Processing a Web Service Response (I) Processing a Web Service Response (II) Summary xix

20 Appendix A: Practices Appendix B: Solutions Appendix C: The Topology Matrix Appendix D: Topology Naming Convention Appendix E: Examples of Data Server Definitions xx