Using ODI for Teradata Dual-Loads

Transcription

1 Purpose This tutorial walks you through the steps that are needed to create Oracle Data Integrator (ODI) EL-T interfaces and packages that will simultaneously load two independent Teradata RDBMS systems configured as dual-active. The source for the data loads is an Oracle database. Time to Complete: Approximately 120 minutes Topics This tutorial covers the following topics: 1. Overview 2. Teradata Dual-Load Architecture using ODI 3. Installation & Setup Prerequisites 4. Setup the ODI Environment 5. Setup the Source and Destination Databases 6. Create Oracle to Flat File Interface 7. Create Interfaces for Scenario 1 8. Create Interfaces for Scenario 2 9. Execute Tests for Scenario Execute Tests for Scenario Summary Note: The screenshots will not reflect the specific environment that you are using. They are provided to give you an idea of where to locate specific functionality in Oracle Data Integrator. 1. Overview A Teradata dual-active system is recommended for high availability of Teradata systems and guarantees business continuity by having the data always available in a secondary system when the primary becomes unavailable. One method of loading a dual-active Teradata system is by using a dual-load approach. Dual-loads perform massive data loads and transforms from the same source into two separate Teradata systems, one of which is designated as Primary and the other as Secondary. Dual-loads can be performed using multiple scenarios, but the two most popular scenarios are: Scenario 1 - Parallel Loads and Transforms on Primary and Secondary Scenario 2 - Load and Transform on Primary and Replicate to Secondary The first scenario is recommended when it is expected that the frequency of updates on the source system will be less and the data volumes involved in synchronization is high. This scenario will provide the best loading performance while guaranteeing the least overhead. The second scenario eliminates the need to run transformations on the secondary system since the data will be loaded and transformed on the primary and then only the changed (and transformed) data will be pushed to the secondary system. This tutorial will cover both scenarios. For more information on the architecture of Teradata dual-loads using ODI, refer to the whitepaper Enabling Teradata Dual Active using Oracle Data Integrator. 2. Teradata Dual-Load Architecture using ODI The dual-load process deals with loading two systems in parallel with the same data.. The loading solution must take into account the fact that either system may be down at any point in time and miss one or several 2010 Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 1

2 loading/synchronization cycles. The loads should be smart enough to catch-up once the system becomes available Scenario 1 - Parallel Loads and Transforms on Primary and Secondary: In this scenario, a first job will extract data from the source Oracle system. Then, two separate processes will load each of the target Teradata systems. Transformations will be performed in parallel on both systems, and execution can be monitored independently on each of the target system. Figure 1: Scenario 1 Parallel Loads and Transforms on Primary and Secondary Extract The extraction is performed only once since reading twice from the source system would be counterproductive. But as the extraction is performed, one of the target systems (or both) could be down. To guarantee the delivery of information, extracted data must first be staged in flat files. Each file extract is numbered with a Batch ID to enable recovery when needed. Each set of extracted files is stored in a directory which is named after the Batch ID. Each table dump will be stored as a file in this directory. The next run creates a new directory where the new files will be created. As soon as both systems have loaded a given sets of files, the files can be discarded and the directory can be deleted or archived. The Extract can be smart enough to implement change data capture from the source system. There are multiple ways of enabling changed data capture using ODI, but for the purpose of this tutorial, we will use a simple timestamp method using which we will stamp a Date/Time field on the rows that are inserted or modified on the source table. The extract process will then store the highest processed timestamp in a variable, and use its value to determine the rows that will be extracted in the next run Load The Load operation will be exactly the same on both systems: we have to perform a traditional ELT operation, using which the files are bulk loaded into a staging area on the target system (using 2010 Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 2

3 TPT/FastLoad) and then transformed to guarantee the best possible performance. As the same load process runs on both the Primary and Secondary systems, the transformation process will be developed only once and executed against multiple systems. ODI provides several mechanisms that can be leveraged to run the same job on different physical environments, via the notion of contexts. Contexts define different physical environments for the same logical processing units. Variables can also be used here as an alternative to contexts to access the different systems dynamically. When loads are not processed because one of the system is non-operational, the scenario must ensure recovery of failed jobs. The solution here is simply to run the process and increment the Batch ID until all available files have been processed Transform A simple transformation during the load process will be demonstrated in this tutorial. By using contexts or variables, the transformations can be developed once and executed on both systems Cleanup The staging files after being loaded to both Teradata systems would need to be cleaned up to save space. This process can be very easily built using ODI and is not discussed in this tutorial Scenario 2 - Load and Transform on Primary and Replicate to Secondary: This scenario will perform loads on the Primary Teradata system from the Oracle database using the steps and packages created in scenario 1. Once the data is loaded into the Primary Teradata system, the scenario 2 will replicate the data from the Primary system to the Secondary Teradata system, instead of the secondary system getting loaded from the Oracle database like scenario 1. The extract and transformations processes created in scenario 1 will be used in this process to load into the Primary Teradata system. One the data is in the primary system, there will be a Replicate package that will replicate the data loaded from the primary system to the secondary system. The method of capturing changes in the Replicate package is to utilize a simple timestamp based mechanism that identifies the inserted/updated rows in the Primary system. The replicate package will pull these rows based on this timestamp, and the last run of the Replicate package. The benefit of scenario 2 is that the transformations will be performed only once, on the Primary Teradata system, as opposed to scenario 1, which will run transformations on the Primary and the Secondary Teradata systems Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 3

4 Figure 2: Scenario 2 - Load and Transform on Primary and Replicate to Secondary 3. Installation and Setup Prerequisites Before you start the tasks, make sure that the following requirements have been met: The system should include the following installed products: o Oracle Database (10g, 11g or 10gXE) o Oracle Data Integrator 10g ( ) o Primary Teradata system o Secondary Teradata system o Oracle Data Integrator 10g ( ) Agent installations on primary Teradata You should have successfully completed the Oracle by Example (OBE) titled Creating and Connecting to ODI Master and Work Repositories before attempting this tutorial. Click here to access this OBE. You should have downloaded the zip file that contains the ODI seed projects and models used in this tutorial. Click here to download the required files. Note 1: Before proceeding further, start the services and components to start the Oracle Database, both the Teradata systems, and Oracle Data Integrator 10g ( ) Agent on the Teradata primary system. Note 2: For the purposes of this tutorial, the ODI software and the source Oracle Database are installed on the same machine. Also, we will be using one ODI Agent to service both Teradata systems. 4. Setup the ODI Environment 4.1. Configure ODI Topology The ODI Topology must be configured to access the Teradata systems, Oracle sources, Flat file sources and the ODI Agents. The ODI Topology defines the Physical Architecture, the Logical Architecture as well as the Contexts that associate a logical architecture with the physical Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 4

5 The following sections provide the steps necessary to modify the Topology Login to the Repository using Topology Manager Click on Start -> All Programs -> Oracle -> Oracle Data Integrator -> Topology Manager from the Start Menu. Login to the repository using the SUPERVISOR account Create Contexts For the purpose of this lab, we will create two contexts identifying the primary and secondary Teradata systems. All loads that run on the primary Teradata system will run under the Primary context and the ones for secondary Teradata system will run under the Secondary context. The extracts from the Oracle source will run under the Global context Create the Primary Context: This context will be associated with the primary Teradata system - TD1. Click on the Contexts tab from the Topology Manger window. Click on the Insert Context icon to get the dialog-box to enter the new context. Note: If the Insert Context icon is not visible, then from the Topology Manager, click on Windows -> Show View -> Contexts Click on the Definition tab and enter the name for the new context as Primary. Leave all other items as default. Click OK to create the primary context Create the Secondary Context: This context will be associated with the secondary Teradata system - TD Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 5

6 Click on the Insert Context icon again to get the dialog-box to enter the new context. Then click on the Definition tab. Enter the name as Secondary. Leave all other items as default. Click OK to create the Secondary Context. The Context pane should look similar to the one below: Configure ODI Agents in the Topology Note: Make sure the ODI Agent is started on the primary Teradata system before proceeding with this step Click on Physical Architecture tab, right-click on Agents and select Insert Agent Enter the below information for the Agent. Name Host TD1_Agent <Teradata_Host> Where <Teradata_Host> is the hostname or IP Address of the primary Teradata system. Leave all other information as default Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 6

7 Click Apply and then click Test and you should see the below message: Click OK to close the message, and OK to close the Agent configuration window. You should end up with the below Agent in the Physical Architecture tab. Note: In this tutorial, we will be configuring only one ODI Agent on the primary Teradata system only. The same agent will handle execution of jobs on the secondary system. Note: The source Oracle database runs on the same system as the ODI software, there is no need to configure an Agent for the source Click on the Logical Architecture tab and right-click Agents and select Insert Logical Agent Enter TD_Agent as the name. Associate TD1_Agent to the Primary and the Secondary contexts. Click OK You should end-up with one Logical Agent as shown below: 2010 Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 7

8 Setup and configure the Teradata systems in the ODI Topology Configure the primary Teradata system. Select the Physical Architecture tab and expand the Technologies folder, right-click on Teradata and select Insert Data Server In the Data Server New window, click on the Definition tab and enter the below information Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 8

9 Name Server (Data Server) User Password TD1 <Teradata_Server> DBC Where <Teradata_Server> is the hostname or IP Address of the Primary Teradata system. Dbc Click the JDBC tab. Click the ellipses ( ) to enter the JDBC Driver information. Select the NCR Teradata JDBC Driver from the drop-down. Click OK on the Drivers dialog box. Important Note: Make sure that you copy the NCR Teradata JDBC Drivers (mainly the terajdbc4.jar, tdgssconfig.jar and tdgssjava.jar files) to the ODI_HOME/oracledi/drivers directory. ODI needs the drivers placed in this location in order to connect to Teradata via JDBC Enter the JDBC URL. The URL is of the form jdbc:teradata://<host>:<port>/<server>, where <host> is the hostname of the Teradata system, <port> the port it s configured to listen on, and <server> is the server name. The below example assumes that the staging tables are created in the financial schema. Note: If you do not have the financial Schema created in the Teradata environments, then you should create it prior to proceeding with the rest of the steps. Alternatively, you could replace the financial schema referenced in this tutorial with the schema of your choice Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 9

10 Click on Test to test the connection. Select the TD1_Agent from the drop-down. Click Test again. You should get a successful connection message like the one below. Click OK, then click OK again to close the Data Server New window. This will add the new Teradata server TD1 to the Physical Architecture Set the physical schema and the work schema for the TD1 connection. In the Properties box that opens, select financial from the Schema (Schema) drop-down and financial from the Schema (Work Schema) drop-down Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 10

11 Click OK. Click OK again when prompted for No Context specified for this Schema. We will create the context in the later steps Configure the Secondary Teradata System. Repeat Steps through in order to configure the secondary Teradata system. Modify the hostname or the IP address of the secondary system as appropriate. Also note that we will be using TD1_Agent to connect to this system. Note: If you wish to configure the secondary Teradata system on a different schema on the primary Teradata system, then replace the financial schema identified in steps through with the schema of your choice. Name Server (Data Server) User Password TD2 <Teradata_Server> DBC Dbc Where <Teradata_Server> is the hostname or IP Address of the Primary Teradata system Create the Teradata Logical Schema. Click on the Logical Architecture tab and expand Technologies -> Teradata. Right-click on Teradata and select Insert Logical Schema Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 11

12 Enter TD as the name. Select TD1.financial physical schema for the Primary context and TD2.financial physical schema for the Secondary context. Leave other selections as default. Click OK. The logical schema TD should be created as shown below Configure the Oracle Data source Physical and Logical Schemas Create the Oracle schema for storing the source tables. This tutorial will use the sales history (SH) schema that comes pre-installed with the Oracle example schemas. SQL> create user SH identified by <password> default tablespace USERS temporary tablespace TEMP; SQL> grant connect, resource to SH; Select the Physical Architecture tab and expand the Technologies folder, right-click on Oracle and select Insert Data Server. In the Data Server New window, click on the Definition tab and enter the below information. Name <Oracle_SID> 2010 Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 12

13 Server (Data Server) User Password <host> SH <password> Where <Oracle_SID> is the SID of the source Oracle system, <host> is the hostname or IP Address of the Oracle system, <password> is the password for SH user Click the JDBC tab. Click the ellipses ( ) to enter the JDBC Driver information. Select the Oracle JDBC Driver from the drop-down. Click OK on the Drivers dialog box Enter the JDBC URL. Click the ellipses ( ) besides the JDBC URL to enter the JDBC URL information. Select the jdbc:oracle:thin@<host>:<port>:<sid> from the drop-down. Where <host> is the server running the Oracle database, <port> is the listener port, and <sid> is the Database SID. Enter the JDBC URL that is appropriate for your environment. The URL used for this tutorial is jdbc:oracle:thin:@xpobi-1-vm:1521:orcl. This means that the Oracle database server is running on xpobi-1-vm, the listener is configured on the default 1521 port, and ORCL is the database SID Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 13

14 Click on Test to test the connection. Select the Local (No Agent) from the drop-down. Click Test again. You should get a Successful connection window like the one below. Click OK, then click OK again to close the Data Server New window. This will add the new Oracle server ORCL to the Physical Architecture Set the Physical Schema and the Work Schema for the ORCL connection. In the Properties box that opens, select SH from the Schema (Schema) drop-down and SH from the Schema (Work Schema) drop-down Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 14

15 Click OK and click OK again when prompted for No Context specified for this Schema. We will create the context in the later steps Create the Oracle Logical Schema. Click on the Logical Architecture tab, expand Technologies -> Oracle. Right-click on Oracle and select Insert Logical Schema. Enter ORCL_SH as the name. Select ORCL.SH as the Physical Schema for Global, Primary and the Secondary contexts. Click OK. The logical schema ORCL should be created as shown below Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 15

16 Configure the Physical and Logical Schemas for the Flat File Source. In this step, we will be modifying the existing FILE_GENERIC source that comes pre-installed in the ODI Topology. Note: If you do not see a FILE_GENERIC source in your tree then add one. For instructions on adding File sources in ODI, refer to Oracle by example Creating ODI Project and Interface: Exporting Relational Table to Flat file. Click here to access this OBE Configure the Flat File Physical Source. Select the Physical Architecture tab and expand the Technologies -> File folder, and double-click on the FILE_GENERIC source In the Data Server New window, click on the Definition tab and enter the below information. Name Server (Data Server) FILE_GENERIC <host> Where <host> is the hostname or IP Address of the Host that will store the files Click on the JDBC tab and make sure that the JDBC Driver and JDBC URL is selected as shown below Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 16

17 Click on Test and select Local (No Agent) for the test. You should get a Successful Connection message displayed. Click OK to clear the message. Click OK to save the File Physical Source Set the physical schema and the work schema for the FILE_GENERIC connection. Double-click on the FILE_GENERIC../demo/file Physical Schema under the FILE_GENERIC source. In the Properties box that opens, enter /tmp for the Schema (Schema) and the Schema (Work Schema) fields. Leave all other fields as defaults Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 17

18 Click OK to save your changes Create the Flat File Logical Source. Click on the Logical Architecture tab, expand Technologies -> File. Double-click on FILE_DEMO_SRC. Note: If you do not see a FILE_DEMO_SRC in your Topology then create one. Refer to the OBE Exporting a flat file to a relational table for more details on creating file sources Select FILE_GENERIC./tmp as the Physical Schemas for Global, Primary and the Secondary contexts. Click OK to save your changes Setup the ODI Designer environment Login to the Repository using ODI Designer. Click on Start -> All Programs -> Oracle -> Oracle Data Integrator -> Designer from the Start Menu. Login to the repository using the SUPERVISOR account Import the seed Project supplied in the zip file. The seed project PROJ_TD_Dual_Load.xml has the following pre-created items: The required Teradata Knowledge Modules for dual-load Variables required by the dual-load examples 2010 Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 18

19 Procedures to create the Oracle Table SRC_SALES_PERSON which is the source table Procedures to create the Teradata table TD_SALES_PERSON & TD_REPL_SALES_PERSON which are the destination tables in Teradata Systems Packages to initialize the Variables that are needed for the ELT Packages to incrementally load into the Source table Procedure to get the max Batch ID that the source extract is currently using Right-click on the blank area in the Projects pane of ODI Designer and select Import Project. Alternatively you can click on the Import Project icon as well Click on ellipses ( ) to select the Projects directory from the supplied zip file. Select the TD_Dual_Load as the project file to import. Click OK to import. After the import the TD_Dual_Load Project tree should expand like shown below Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 19

20 Import the Teradata, Oracle and File models. The supplied Teradata, Flat File and Oracle models contains the below objects: SRC_SALES_PERSON: This is the source table for the dual-load and resides in the Oracle Database. The table contains a DATE_UPDATED column that will be used as the basis of the Changed Data Capture process. DL_VARIABLES: This table stores the value of the latest Batch ID that is associated with the Oracle extract. The purpose of this is to pass the variable values between different ODI contexts. TD_SALES_PERSON: This is the destination table that exists in the Primary and Secondary Teradata Systems. This table contains a derived column FULL_NAME that is not present in the Source and a simple ETL map will derive this value Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 20

21 TD_REPL_SALES_PERSON: This is the destination table that exists in Secondary Teradata System. This table will be replicated from the Primary Teradata TD_SALES_PERSON table when performing dual-loads using scenario 2. File_Demo_Src: This is the intermediary Flat File that will be created to store the Oracle source table SRC_SALES_PERSON. Click on the Models tab of ODI Designer. Right-click on the blank area in the Models pane and select Import Model Folder. Alternatively you can click on the Import Model Folder icon as well Click on ellipses ( ) to select the Models directory from the supplied zip file. Select the Flat Files, Oracle and Teradata as the models to import holding the CTRL key. Click OK to start the import. After the import is done, the model pane should expand as shown below Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 21

22 5. Setup the Source and Destination Databases In this step, we will create the source and the destination Database tables using the supplied ODI Procedures. We will also run the ODI Procedures that will populate the source table with initial data. We also need to run ODI Packages to initialize the variables used by ODI Interfaces and Packages Create Oracle tables and populate with seed data. The procedure Create_Oracle_Tables has been supplied with the seed project to create and populate the Oracle schema. Follow the below steps to execute the supplied procedure In the ODI Designer Projects pane, expand TD_Dual_Load -> 1. Setup -> Procedures. Right-click on Create Oracle Tables and select Execute Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 22

23 Select the Global context, and Local (No Agent) agent. Click OK to execute the procedure. The Oracle physical schema ORCL_SH has been associated with the Global context. You should see a Session started message. Click OK to acknowledge the message Verify the execution using ODI Operator. Click on the Operator icon from ODI Designer menu. In the Session List pane, expand Date -> Today and you should see a successful execution of the Create Oracle Tables procedure Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 23

24 5.2. Create Teradata tables. The procedure Create_Teradata_Tables has been supplied with the seed project to create the Teradata schema. Follow the below steps to execute the supplied procedure In the ODI Designer Projects pane, expand TD_Dual_Load -> 1. Setup -> Procedures. Right-click on Create Teradata Tables and select Execute. Select the Primary context, and TD_Agent agent. Click OK to execute the procedure. The primary Teradata physical schema TD1 has been associated with the Primary context. You should see a Session started message. Click OK to acknowledge the message Execute the Create Teradata Tables procedure again. And this time select the Secondary context and TD_Agent agent to create the Teradata schema on the secondary Teradata system TD Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 24

25 Click OK to execute the procedure. The secondary Teradata physical schema TD2 has been associated with the Secondary context. Click OK to acknowledge the Session Started message Verify the execution using ODI Operator. Click on the Operator icon from ODI Designer menu. In the Session List pane, expand Date -> Today and you should see two successful executions of the Create Teradata Tables procedures Initialize Variables used by ODI. Execute the Package Initialize Variables to initialize variables used by ODI mappings. The ODI Interfaces and Packages will depend upon seed values of these variables and skipping this step will lead to errors during execution of Interfaces and Packages later In the ODI Designer Projects pane, expand TD_Dual_Load -> 1. Setup -> Package. Right-click on Initialize Variables and select Execute Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 25

26 Select the Global context and Local (No Agent) agent. Click OK. This will execute the package Initialize Variables against the Oracle SH Schema. Click OK to acknowledge the Session Started message. You need to repeat this step and execute the same package against the primary and secondary Teradata sources. This you would do by repeating the previous step for each Primary and Secondary contexts and execute the package against the TD_Agent agent Verify the execution using ODI Operator. Expand Date -> Today and you should three successful executions of the Initialize Variables package Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 26

27 6. Create Oracle to Flat File Interface In this section, we will create ODI Interfaces and Packages that will extract the data out of the Oracle source and into a flat file. This step is common for both the Teradata dual-load scenarios. The flat file is staged in a directory that is created when the extract process runs, and is identified uniquely by the Batch ID of the extract process. This mechanism allows us to run the extracts independently of the Teradata import process hence making the extract process completely asynchronous with the loads. The extracts out of the Oracle SRC_SALES_PERSON table looks at the new rows inserted or modified since the time the last extract was run. The changed data capture mechanism is accomplished by using the DATE_UPDATED column on the SRC_SALES_PERSON table. The column is updated at every inserts or updates at the source database. At the end of the Extract process, there will be three files created in the directory SRC_SALES_PERSON.txt which is the text dump of SRC_SALES_PERSON table, the Primary.txt and Secondary.txt files which are used as a flags to indicate whether or not the dump file was loaded into the Primary or the Secondary Teradata Systems respectively. These files are deleted after the dump is successfully loaded into its respective destination Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 27

28 6.1. Create a New Folder for the Extract process In the ODI Designer Projects pane, right-click on TD_Dual_Load and select Insert Folder Enter 2. Extract as the name and click OK Create an Oracle Table to Flat file Interface Expand the newly created 2. Extract folder and right-click on Interfaces and select Insert Interface. We will be creating the Interface to extract an Oracle table to a Flat file Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 28

29 Enter Oracle to File in the Name field and set the Optimization Context as Global. Check the Staging Area Different From Target and select ORCL_SH to indicate that the staging area for any temporary objects needed by the ELT is going to be the ORCL_SH schema. Click on the Diagram tab. And click OK if prompted for the below message Create the ELT map between SRC_SALES_PERSON table in ORCL_SH schema to SRC_SALES_PERSON Flat file. In the left pane of ODI Designer, click on Models tab. Expand the Oracle -> ORCL_SH model folder Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 29

30 Drag and drop the SRC_SALES_PERSON table to the Sources section of the Diagram as shown below. Next, expand the model folder Flat Files -> File_Demo_Src. Drag and drop the SRC_SALES_PERSON file from File_Demo_Src to the Target Datastore section of the diagram. Click Yes when prompted for Automatic Mapping as shown below Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 30

31 You should see Source and the Target columns mapped as shown in the below diagram Create a filter on DATE_UPDATED column. This is needed to implement the change data capture logic for extracting only the newly inserted rows or modified rows from SRC_SALES_PERSON table. In order to create a filter, drag and drop the DATE_UPDATED column out of the SRC_SALES_PERSON table in the Sources section and into the blank area on the Sources pane. Click on the filer icon. Make sure the filter is selected to be active by checking the Active Filter box. Click on the Implementation tab and Enter: SRC_SALES_PERSON.DATE_UPDATED BETWEEN TO_TIMESTAMP('#LastRefresh','MM/DD/YYYY HH24:MI:SS.FF') AND TO_TIMESTAMP('#CurrentTimestamp','MM/DD?YYYY HH24:MI:SS.FF') 2010 Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 31

32 Click on the Database icon to check for syntactical errors in the filter expression. You should see the below message that says that the SQL is valid for the DBMS. Click OK. Then click on Errors to check for errors in the mapping. You should see the below text box stating that there were no errors detected in the diagram. Click OK to close the text box Click on the Flow tab. This is where we will be assigning the appropriate knowledge modules for the ELT. Click on Target (FILE_GENERIC) table. In the Properties section below, make sure the following values are set: IKM INSERT TRUNCATE GENERATE_HEADER IKM SQL to File Append Yes Yes No 2010 Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 32

33 Click OK to save the Oracle to File Interface Create a package that will call the Oracle to File Interface Create a new package. In the Projects pane of ODI Designer, expand TD_Dual_Load -> 2. Extract. Right-click on Packages and select Insert Package. Click on the Definition tab of the Package: New window and enter Extract as the name Create the package flow. Click on the Diagram tab of the Package: New window. On the Projects pane, expand TD_Dual_Load -> Variables folder. Drag and drop BatchID variable to the Diagram pane as shown below Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 33

34 Similarly, drag and drop PathID, FilePath, LastRefresh, CurrentTimestamp, BatchID and LastRefresh from the Variables folder to the Diagram pane. Note that some of the variables are used multiple times in the diagram, and that is not a typo. Next, drag and drop Oracle to File Interface from TD_Dual_Load -> 2.Extract -> Interfaces folder, and Update MaxID from TD_Dual_Load -> 1.Setup -> Procedures folder. Next, add Files -> OdiMkdir, File -> OdiOutFile (twice) from the Toolbox pane. The way to add items from the Toolbox is to first click on the tool you want to add, and then click again on the Diagram pane to add the tool to the diagram. When you are done adding tools, then click on the adding. icon to stop Once you are done adding all the elements, arrange the icons in the diagram such that they look similar to the picture below Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 34

35 Add the Next steps on Success ( ) arrows between all the steps. It is important to ensure the flow is exactly as it looks in the below diagram. Click on the selection pointer before continuing with the next steps Set properties for each step. We will start with BatchID (top left in the diagram) which is the first step that will be executed and just go along with the next step in the sequence Set properties for BatchID. Highlight the BatchID step in the diagram and click on the General tab of the Properties section. Make sure you set the properties exactly as shown below Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 35

36 Set properties for PathID. Highlight the PathID step in the diagram and set the properties as shown below Set properties for FilePath. Highlight the FilePath step in the diagram and set the properties as shown below Set properties for OdiMkDir1. Highlight the OdiMkDir1 step in the diagram and set the properties as shown below Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 36

37 Set properties for LastRefresh. Highlight the LastRefresh step in the diagram and set the properties as shown below Set properties for CurrentTimestamp. Highlight the CurrentTimestamp step in the diagram and set the properties as shown below Set properties for Oracle to File. Highlight the Oracle to File step in the diagram and set the properties as shown below Set properties for OdiOutFile2. Highlight the OdiOutFile2 step in the diagram and enter the Target File name. Make sure you hit Enter to make your values persist in text boxes Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 37

38 Click on the ellipses ( ) next to the Text field. This will open the Expression Editor tool. In the top box, enter the text Primary as shown below. Click OK to save your expression Set properties for OdiOutFile3. Highlight the OdiOutFile3 step in the diagram and enter the properties as shown below. Follow the instructions mentioned in the previous step Set properties for Update MaxID. Highlight the Update MaxID step in the diagram and set the properties as shown below Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 38

39 Set properties for BatchID. Highlight the second BatchID step in the diagram and set the properties as shown below. This time we will refresh the BatchID variable Set properties for LastRefresh. Highlight the second LastRefresh step in the diagram and set the properties as shown below. This time we will set the LastRefresh variable to the CurrentTimestamp Once done setting the properties of all the steps, click OK to create the Extract package. 7. Create E-LT Interfaces for Scenario 1 In this section, we will create ODI Interfaces and Packages that will load the data from the flat file that was extracted in the prior steps, into the Teradata systems. The Load process utilizes the Batch ID of the extract process and figures out how many batches of the extract it has to load into the Teradata system in order to keep it in sync with the source system. This process is completely asynchronous with the extracts and can be run at different frequencies altogether. Also if the load process fails for any reason, it can be re-executed and the load will pick up all the missed extracts since the last successful run Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 39

40 The Primary.txt and Secondary.txt files that will be created by the extract process for each Batch ID, will be deleted by the load process upon a successful load of that batch. The existence (or the non-existence) of these files can utilized in the cleanup process to indicates which batches have been consumed by the Primary and the Secondary Teradata systems and are potential candidates for cleanup. If an extract file has been loaded in both the Primary and the Secondary Teradata system, then it is no longer needed and can be deleted or archived. We will be developing one Interface and one Package to perform loads into both Teradata systems. The way we will accomplish this is by associating different Contexts to each system, and executing the Load Package by associating it to a particular Context at runtime Create a New Folder for the Load process In the ODI Designer Projects pane, right-click on TD_Dual_Load and select Insert Folder Enter 3. Load as the name and click OK Create a Flat file to Teradata Interface Expand the newly created 3. Load folder and right-click on Interfaces and select Insert Interface. We will be creating the Interface to load a Flat file to Teradata Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 40

41 Enter File to Teradata in the Name field and set the Optimization Context as Primary. Check the Staging Area Different From Target and select TD from the drop down. Then Uncheck Staging Area Different From Target. Your Interface: NEW box should look like the one below. Click on the Diagram tab. And click OK if prompted for the below message Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 41

42 Create the ELT map to load SRC_SALES_PERSON Flat file into TD_SALES_PERSON Table in Teradata. In the left pane of ODI Designer, click on Models tab. Expand the Flat Files -> File_Demo_Src model folder. Drag and drop the SRC_SALES_PERSON Flat file to the Sources section of the Diagram as shown below. Next, expand the model folder Teradata -> TD_FINANCIAL. Drag and drop the TD_SALES_PERSON file to the Target Database section of the diagram. Click Yes when prompted for Automatic Mapping as shown below Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 42

43 You should see Source and the Target columns mapped as shown in the below diagram. The columns that should not be mapped yet are the DATE_UPDATED and FULL_NAME. We will be creating the Transformations for those columns in the next steps Create the Transformations for FULL_NAME and DATE_UPDATED. Click on FULL_NAME column in the Target Datastore pane. You should see a Mapping: FULL_NAME sub-window towards the bottom. Click on the Implementation tab and enter the below expression in the text box: TRIM(SRC_SALES_PERSON.FIRST_NAME) ' ' TRIM(SRC_SALES_PERSON.LAST_NAME) Check the Active Mapping box. Select Staging Area for the Execute On option. This will ensure the expression will be evaluated at the staging location. You window should be similar to the one below Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 43

44 Click on Errors and make sure you get No Errors detected in your diagram message returned. Click OK Click on DATE_UPDATED column in the Target Datastore pane. You should see a Mapping: DATE_UPDATED sub-window towards the bottom. Click on the Implementation tab and enter the expression CURRENT_TIMESTAMP in the text box and check the Active Mapping box. Select Staging Area for the Execute On option. This will ensure the expression will be evaluated at the staging location. You window should be similar to the one below Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 44

45 Click OK. Then click on Errors to check for errors in the mapping and make sure there are none. Click OK to close the text box Click on the Flow tab of the Interface: NEW window. This is where we will be assigning the appropriate knowledge modules for the ELT. Click on SS_0 (1 FILE_GENERIC) object on the left of the flow. In the properties section below, make sure the following values are set. Leave all other values as default. The TTU_PATH property of the KM needs to be set to blanks. Note: Ensure you hit Enter after making changes to each field otherwise your changes will not persist. LKM LKM File to Teradata (TTU) - DL DL_DIRECTORY #TD_Dual_Load.FilePath//<%=odiRef.getSession("CONTEXT_NAME")%>_DL_Log 2010 Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 45

46 TTU_PATH Click on Target+Staging Area (TD1) object on the right of the flow. In the properties section below, make sure the following values are set. Leave all other values as default. IKM FLOW_CONTROL COLLECT_STATS IKM Teradata Control Append No No 2010 Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 46

47 Click on the Controls tab and check that the Control KM is <Undefined> for this load. Click OK to save the File to Teradata Interface Create a package that will call the File to Teradata Interface Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 47

48 Create a new package. In the Projects pane of ODI Designer, expand TD_Dual_Load -> 3. Load. Right-click on Packages and select Insert Package. Click on the Definition tab of the Package: New window and enter Load as the name Create the package flow. Click on the Diagram tab of the Package: New window. On the Projects pane, expand TD_Dual_Load -> Variables folder. Drag and drop CurrentID variable to the Diagram pane as shown below Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 48

49 Similarly, drag and drop MaxID, PathID, FilePath, CurrentID and CurrentID (again), from the Variables folder to the Diagram pane. Note that some of the variables are used multiple times in the diagram, and that is not a typo. Next, drag and drop File to Teradata Interface from TD_Dual_Load -> 3.Load -> Interfaces folder. Next, add Files -> OdiFileWait, File -> OdiFileDelete from the Toolbox pane. The way to add items from the Toolbox is to first click on the tool you want to add, and then click again on the Diagram pane to add the tool to the diagram. When you are done adding tools, then click on the adding. icon to stop Once you are done adding all the elements, arrange the icons in the diagram such that they look similar to the picture below Add the Next steps on Success ( ) arrows between all the steps. It is important to ensure the flow is exactly as it looks in the below diagram Set properties for each step. We will start with CurrentID (bottom left in the diagram) which is the first step that will be executed and just go along with the next step in the sequence Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 49

50 Set properties for CurrentID. Highlight the CurrentID step in the diagram and click on the General tab of the Properties section. Make sure you set the properties exactly as shown below Set properties for MaxID. Highlight the MaxID step in the diagram and set the properties as shown below Set properties for PathID. Highlight the PathID step in the diagram and set the properties as shown below Set properties for FilePath. Highlight the FilePath step in the diagram and set the properties as shown below Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 50

51 Set properties for OdiFileWait 1. Highlight the OdiFileWait 1 step in the diagram and set the properties as shown below. Directory Filename Mask \tmp\#td_dual_load.filepath <%=odiref.getsession("context_name")%>.txt Set properties for File to Teradata. Highlight the File to Teradata step in the diagram and set the properties as shown below Set properties for OdiFileDelete 2. Highlight the OdiFileDelete 2 step in the diagram and set the properties as shown below. Directory Filename Error if no file found \tmp\#td_dual_load.filepath <%=odiref.getsession("context_name")%>.txt Yes 2010 Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 51

52 Set properties for CurrentID. Highlight the CurrentID step in the diagram that follows OdiFileDelete 2 and set the properties as shown below Set properties for CurrentID. Highlight the CurrentID step in the diagram that follows the previous step s CurrentID and set the properties as shown below. Type Operator <= Value Evaluate Variable #TD_Dual_Load.MaxID Once done setting the properties of all the steps, click OK to create the Load package Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 52

53 8. Create E-LT Interfaces for Scenario 2 Using ODI for Teradata Dual-Loads In this section, we will create ODI Interfaces and Packages that will replicate data from TD1 to TD2 Teradata systems using the approach outlined in scenario 2. The Replicate process will pull out changed data from the TD_SALES_PERSON table from the TD1 Teradata system and push the transactions to the TD_REPL_SALES_PERSON table on TD2 Teradata system. For the change capture process, the E-LT interfaces will look at the DATE_UPDATED column of the TD_SALES_PERSON table and only pull rows that were modified or inserted since the last run of the Replicate process. This process is completely asynchronous with the extracts and the loads processes, and can be run independently and at different frequencies. Also, if the Replicate process due to the unavailability of Teradata TD2 system, the process can be rerun and all the rows that were missed since the last successful run will be replicated in this run Create a New Folder for the Replicate process In the ODI Designer Projects pane, right-click on TD_Dual_Load and select Insert Folder. Enter 4. Replicate as the name and click OK Create an Interface to replicate from TD1 to TD Expand the newly created 4. Replicate folder and right-click on Interfaces and select Insert Interface. We will be creating the Interface to replicate the TD_SALES_PERSON on TD1 Teradata system to TD_REPL_SALES_PERSON on TD2 Teradata system. Enter Replicate TD1 to TD2 in the Name field and set the Optimization Context as Secondary. Check the Staging Area Different From Target and select TD from the drop down. Your Interface: NEW box should look like the one below Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 53

54 Click on the Diagram tab. And click OK if prompted for the below message Create the E-LT map to replicate TD_SALES_PERSON from TD1 Teradata system to TD_REPL_SALES_PERSON in TD2 Teradata system. In the left pane of ODI Designer, click on Models tab. Expand the Teradata -> TD_SALES_PERSON model folder. Drag and drop the TD_SALES_PERSON Table to the Sources section of the Diagram as shown below. Next, drag and drop the TD_REPL_SALES_PERSON Table to the Target Datastore section of the Diagram as shown below Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 54

55 Click Yes when prompted for Automatic Mapping. You should see Source and the Target columns mapped appropriately based on the column names Click on each column on TD_REPL_SALES_PERSON and make sure all the columns are set to execute on the Staging Area. You can start by TD_REPL_SALES_PERSON.SALES_PERSON_ID as shown below. Repeat this step for the remaining columns FIRST_NAME, LAST_NAME, FULL_NAME, DATE_HIRED and DATE_UPDATED Set the appropriate Contexts for the TD_SALES_PERSON source, TD_REPL_SALES_PERSON destination. Click on TD_SALES_PERSON in the Source section. In the properties box, set the Context as Primary. This is needed to force the source to go against the Teradata source that is setup against the Primary context, which in this case is the TD1 Teradata system Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 55

56 Click on TD_REPL_SALES_PERSON in the Target Datastore section. In the properties box, set the Context as Execution Context. When we execute this interface, the Secondary context will be used and the Execution Context will be associated with the TD2 Teradata at execution time. Also, make sure that the Update Key is set to PK_TD_REPL_SALES_PERSON. This is needed for the Teradata loading KMs when performing updates Create a filter on DATE_UPDATED column on TD_SALES_PERSON. This is needed to implement the change data capture logic for extracting only the newly inserted rows or modified rows from TD_SALES_PERSON table from TD1 source system. In order to create a filter, drag and drop the DATE_UPDATED column out of the SRC_SALES_PERSON table in the Sources section and into the blank area on the Sources pane. Click on the filer icon. Make sure the filter is selected to be active by checking the Active Filter box. Set the Execute On to Source. This means that the filter will be applied on the source Teradata system (TD1). Click on the Implementation tab and Enter: TD_SALES_PERSON.DATE_UPDATED between cast('#lastreplrefresh' as timestamp(2)) AND cast('#currentrepltimestamp' as timestamp(2)) 2010 Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 56

57 Click on the Database icon to check for syntactical errors in the filter expression. You should see the below message that says that the SQL is valid for the DBMS. Click OK. Then click on Errors to check for errors in the mapping. You should see the below text box stating that there were no errors detected in the diagram. Click OK to close the text box Assign the appropriate Knowledge Modules to the Flow. Click on the Flow tab. This is where we will be assigning the appropriate knowledge modules for the E-LT. Click on TD_SALES_PERSON table. In the Properties section below, make sure the following values are set: 2010 Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 57

58 LKM DELETE_TEMPORARY_OBJECTS Yes LKM SQL to SQL Click on TD_REPL_SALES_PERSON table. In the Properties section below, modify the below fields: IKM SYNC_JRN_DELETE FLOW_CONTROL COLLECT_STATS IKM Teradata Incremental Update No No No 2010 Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 58

59 Click OK to save the Replicate TD1 to TD2 Interface Create a package that will call the Replicate TD1 to TD2 Interface Create a new package. In the Projects pane of ODI Designer, expand TD_Dual_Load -> 4. Replicate. Right-click on Packages and select Insert Package. Click on the Definition tab of the Package: New window and enter Replicate as the name Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 59

60 Create the package flow. Click on the Diagram tab of the Package: New window. On the Projects pane, expand TD_Dual_Load -> Variables folder. Drag and drop LastREPLRefresh variable to the Diagram pane as shown below Similarly, drag and drop CurrentREPLTimestamp and LastREPLRefresh (again) from the Variables folder to the Diagram pane. Next, drag and drop Replicate TD1 to TD2 Interface from TD_Dual_Load -> 4. Replicate -> Interfaces folder. Once you are done adding all the elements, arrange the icons in the diagram such that they look similar to the picture below Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 60

61 Add the Next steps on Success ( ) arrows between all the steps. It is important to ensure the flow is exactly as it looks in the below diagram Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 61

62 Set properties for each step. We will start with LastREPLRefresh (top left in the diagram) which is the first step that will be executed and just go along with the next step in the sequence Highlight the LastREPLRefresh (top left in the diagram) step in the diagram and click on the General tab of the Properties section. Make sure you set the properties exactly as shown below Highlight the CurrentREPLTimestamp step in the diagram and set the properties as shown below Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 62

63 Highlight the Replicate TD1 to TD2 step in the diagram and set the properties as shown below Highlight the LastREPLRefresh (bottom left in the diagram) step in the diagram and set the properties as shown below Once done setting the properties of all the steps, click OK to create the Replicate package. 9. Execute Tests for Scenario 1 In this step, you will execute the packages and interfaces created in the previous steps and demonstrate the Teradata dual-load scenario as depicted in scenario 1. The test cases will involve executing the Extract process multiple times. In between the extracts, you will perform an incremental load on the Oracle source. The idea here is to demonstrate that the updates or inserts in the source system will get propagated to the destination using the change data capture mechanisms built into the E- LT maps. The Load package will be executed after the extracts are completed. You will also be performing sanity checks to make sure the test is successful. In a production implementation, the execution of these packages would be automated through the use of ODI Scenarios and a scheduler Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 63

64 9.1. Execute the Extract Package This step is going to run the Extract package for the first time in this tutorial, so this run will extract all the rows from the Oracle database. In the Projects pane of ODI Designer, expand TD_Dual_Load -> 2. Extract -> Packages. Right-click on the Extract Package and select Execute. This Package will run against the SH schema in Oracle, select the Global context, and Local (No Agent) agent. Click OK. You should see a Session started message. Click OK Repeat the above step and execute Extract again. This time the Extract process should not extract any rows since the source data was not modified. We will verify this in the later steps Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 64

65 Login to the Repository using Operator. Click on Start -> All Programs -> Oracle -> Oracle Data Integrator -> Operator from the Start Menu. Or, if you are already logged into Designer or Topology Manager, then click on the Operator icon. In the Session List pane, expand Date -> Today and you should see two successful executions of the Extract Package Using the operating system File Explorer tool, browse to the \tmp directory on the Teradata Primary machine. The \tmp directory should have two sub-directories created underneath as shown below. Each sub-directory represents a successful execution of the extract process Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 65

66 Browse to the \tmp\fileset directory and you should see three files created with the names Primary.txt, Secondary.txt and SRC_SALES_PERSON.txt. The SRC_SALES_PERSON.txt contains the flat file extract of the SRC_SALES_PERSON table from the Oracle SH schema. The presence of Primary.txt and Secondary.txt files indicate that the extract was not yet loaded into any of the Teradata systems. When the extract gets successfully loaded by the Load package to the Teradata systems, these files will get deleted Similar to the above step, browse to the \tmp\fileset directory and you should see the same three files as in the previous step. Notice that the SRC_SALES_PERSON.txt file has zero bytes for this run. This is because when the extract was run the second time, it did not pull any changes from the source table because the source table was not loaded or updated between two consecutive extracts Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 66

67 Note: Look at the content of the files. The SRC_SALES_PERSON.txt for the first extract should have a total of 5 rows, since the source table on the Oracle database had 5 rows Execute the Load Package on the Primary and Secondary Teradata systems This step will run the Load package. The Load package will load these files to the Teradata systems, and also perform simple transformations. In the Projects pane of ODI Designer, expand TD_Dual_Load -> 3. Load -> Packages. Right-click on the Load Package and select Execute. To run the load into the Teradata TD1 Primary system, select the Primary context, and TD_Agent agent in the Execution dialog box. Click OK. Click OK again when you get a confirmation that the session was started Repeat the above step again but execute against the Teradata TD2 Secondary system. In the Execution dialog box, select the Secondary context, and TD_Agent agent as shown below Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 67

68 Verify that the two Loads were successfully completed using ODI Operator Browse to the \tmp\fileset directory on the Teradata Primary system. The directory should have two sub-directories (Primary_DL_Log and Secondary_DL_log) created as shown below. Each sub-directory stores the log files and scripts of the load process for each Teradata system. Also, note that the Primary.txt and Secondary.txt files are non-existent since the SRC_SALES_PERSON.txt dump was loaded into both Teradata systems Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 68

69 Repeat the above step and similarly verify the sub-directories and files present in the \tmp\fileset directory. Note: Browse the Primary_DL_Log and Secondary_DL_log sub-directories and look at the log files generated by the load process Perform an incremental load on the Oracle source and run an Incremental Extract Run the supplied incremental load into the SRC_SALES_PERSON table in Oracle SH schema. The load will insert 5 additional rows into the table. In the Projects pane of ODI Designer, expand TD_Dual_Load -> 1. Setup -> Procedures. Right-click on the Incremental Load Oracle 1 Procedure and select Execute. Select the Global context and Local (No Agent) agent. Click OK. This will execute the package against the Oracle database SH Schema. Click OK to acknowledge the Session Started message Verify that the Incremental load was successfully completed through ODI Operator Get the row count of SH.SRC_SALES_PERSON table from Oracle. You can use SQL*Plus or any SQL Query tool like SQL Developer to login to Oracle. Run the below statement to verify the row counts. The table should have a total of 10 rows. SQL> select count(*) from SH.SRC_SALES_PERSON; 2010 Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 69

70 Execute the Extract Package to get the incremental changes from SRC_SALES_PERSON table in the SH schema into the intermediary flat file that will be loaded into the Teradata systems in the next steps. In the Projects pane of ODI Designer, expand TD_Dual_Load -> 2. Extract -> Packages. Rightclick on the Extract Package and select Execute. This Package will run against the SH schema in Oracle, select the Global context, and Local (No Agent) agent. Click OK. Click OK to acknowledge the Session Started message Verify that the Extract Package was successfully executed by using ODI Operator Verify the rows extracted in the flat file. Using a File Explorer tool, browse to the \tmp\fileset directory on the Teradata Primary machine. Double-click the SRC_SALES_PERSON.txt file. You should see 5 rows that were loaded into the source Oracle table in the previous step Run the Load package to load the extracted file to the Primary and Secondary Teradata systems Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 70

71 In the Projects pane of ODI Designer, expand TD_Dual_Load -> 3. Load -> Packages. Right-click on the Load Package and select Execute. To run the load into the Teradata TD1 Primary system, select the Primary context, and TD_Agent agent in the Execution dialog box. Click OK. You should get a confirmation that the session was started. Click OK to confirm Repeat the above step again but execute against the Teradata TD2 Secondary system. So, in the Execution dialog box, select the Secondary context, and TD_Agent agent Verify that the Load Package was successfully executed by using ODI Operator Verify the incremental loads in Teradata systems by SQL. The TD_SALES_PERSON table in both Teradata systems should have a total of 10 rows as shown below, 5 of which were loaded by the last execution of the Load process Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 71

72 10. Execute Tests for Scenario 2 It is important that you execute the tests for scenario 1 prior to running the scenario 2 tests. This scenario builds on top of the tests for the previous scenario. The scenario 1 should have by now, loaded 10 rows altogether on the Oracle database, and also loaded these 10 rows along with transformations, to the two Teradata systems Primary and Secondary. The test scenarios in scenario 2 will involve executing the Replicate process multiple times to replicate data from the TD_SALES_PERSON table in TD1 to TD_REPL_SALES_PERSON table in TD2. In between running two Replicate packages, we will perform an incremental load on the Oracle source that will load 5 additional rows in the SRC_SALES_PERSON table. The idea here is to demonstrate that the updates or inserts in the source system will get propagated to the destination using the change data capture mechanisms. In a production implementation, the execution of these packages would be automated through ODI Scenarios and scheduler Execute the Replicate Package Verify the number of rows in the TD_REPL_SALES_PERSON table on the Teradata TD2 system using SQL. There should be a total of 0 rows in that table since we have not replicated any data yet Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 72

73 This step is going to run the Replicate Package for the first time in this tutorial, so this run will perform a full load of the TD_REPL_SALES_PERSON table on TD2 from TD_SALES_PERSON on TD1. In the Projects pane of ODI Designer, expand TD_Dual_Load -> 4. Replicate -> Packages. Right-click on the Replicate Package and select Execute. This Package will run against the TD2 Teradata system as we had discussed when creating the Replicate package, select the Secondary context, and TD_Agent agent. Click OK. You should get a confirmation that the session was started. Click OK to confirm Verify that the Replicate Package was successfully executed by using ODI Operator Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 73

74 Verify the number of rows loaded in the TD_REPL_SALES_PERSON table on the Teradata TD2 system using the below SQL. There should be a total of 10 rows in that table, the 10 being loaded by the Replicate package that ran in the previous step Perform an incremental load on the Oracle Source and run the incremental load process on TD1 using scenario 1 and on TD2 using scenario In the Projects pane of ODI Designer, expand TD_Dual_Load -> 1. Setup -> Procedures. Right-click on the Incremental Load Oracle 2 procedure and select Execute. Select the Global context and Local (No Agent) agent. Click OK Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 74

75 This step will execute the package against the Oracle database SH Schema and insert 5 additional rows into the SRC_SALES_PERSON table. Click OK to acknowledge the Session Started message Verify that the Incremental load was successfully completed through ODI Operator Get the row count of SH.SRC_SALES_PERSON table from Oracle. The table should now have a total of 15 rows Execute the Extract Package to get the incremental changes from SRC_SALES_PERSON table in the SH schema into the intermediary flat file that will be loaded into the Teradata systems. In the Projects pane of ODI Designer, expand TD_Dual_Load -> 2. Extract -> Packages. Rightclick on the Extract Package and select Execute. This Package will run against the SH schema in Oracle, select the Global context, and Local (No Agent) agent. Click OK. Click OK to acknowledge the Session Started message Verify that the Extract Package was successfully executed by using ODI Operator Run the Load package to capture incremental changes on the source into the Primary Teradata system. In the Projects pane of ODI Designer, expand TD_Dual_Load -> 3. Load -> Packages. Right-click on the Load Package and select Execute. To run the load into the Teradata TD1 Primary system, select the Primary context, and TD_Agent agent in the Execution dialog box Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 75

76 Click OK. You should get a confirmation that the session was started. Click OK to confirm Verify that the Load Package was successfully executed by using ODI Operator Run the Replicate Package to push changes from TD_SALES_PERSON on TD1, to TD_REPL_SALES_PERSON table on TD2. In the Projects pane of ODI Designer, expand TD_Dual_Load -> 4. Replicate -> Packages. Rightclick on the Replicate Package and select Execute. Select the Secondary context, and TD_Agent agent. Click OK. Click OK again when you see a Session started message Verify that the Replicate Package was successfully executed by using ODI Operator Verify the number of rows loaded to the TD_SALES_PERSON on TD1. There should be a total of 15 rows, the last 5 being loaded by the Replicate package from the previous step Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 76

77 Verify the number of rows loaded to the TD_REPL_SALES_PERSON. There should be a total of 15 rows as well. 11. Summary In this tutorial, we have successfully demonstrated ODI s dual-load capability into Teradata, using the two scenarios that the customers most commonly use for Teradata dual-active configurations Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 77