IBM Replication Products and Strategies in Data Warehousing. Beth Hamel

Transcription

1 IBM Replication Products and Strategies in Data Warehousing Beth Hamel

2 Agenda This presentation introduces many of the tools and techniques that comprise the IBM solution for enterprise data warehousing through the story of a fictional company - Way Cool!! Airlines

3 Way Cool!! Airlines - Customer Background Company profile: Large airline corporation with a long tradition of great customer service including a well used loyalty program Recently has acquired 2 other airline companies Database profile: Very large databases IMS is the primary OLTP database DB2 for OS/390 used for a growing number of applications, including the customer loyalty program IMS data currently extracted to DB2 as a nightly batch operation

4 Way Cool!! Airlines - Business Challenges Nightly extracts take too long Newly acquired businesses have introduced OLTP databases on DB2 for OS/390 and Oracle Need a consolidated loyalty program database must include and consolidate all customer data highly desirable to obtain consistent data across the business must keep historical data needs to be enabled for e-business need to provide a basis for ad-hoc query and analytical applications must be maintained by fully automated processes nightly processing - changed data only

5 Way Cool!! Airlines - Desired Configuration OLTP Systems Oracle 8.1 IMS V6 DB2 V5 OS/390 OS/390 Solaris Server Operational Data Store Data Warehouse e-business servers

6 IBM Data Replication Components Full Refresh Source Data DB2 Family DataPropagator Relational Capture DB2 Update Staging Tables NotesPump Apply Notes Apply DB2 DataJoiner Multivendor Access Targets Multi-vendor Update DB2 Family Multi-vendor Capture Triggers DataPropagator NonRelational Full Refresh Capture IMS IMS, VSAM, Flat Files, Other DataRefresher Lotus Notes

7 IBM Data Replication for IMS MVS/ESA IMS Database IMS/ESA IMS MPP IMS BMP IMS IFP IMS BATCH CICS DBCTL IMS Log Save Changes DL/I Update Capture Changes IMS Async.. Data Capture Exit IMS Application Program DPropNR Selector PRDS DPropNR Receiver DB2 ON MVS DPropNR mapping logic Propagate Data SQL Statements DB2 Tables IMS data is captured on the IMS OLDS as 9904 log records after an EXIT=(*,LOG) keyword has been added and made active in the appropriate DBDs

8 Hierarchical to Relational Mapping IMS Segment Field Segment Occurrence Parent/Child Relationship DB2 Table Column Row RIR (Referrential Integrity Relationships) Seg A Seg C Seg E Key A Table E Key C Key E Data E DPROPNR supports Mapping of multiple IMS DB segments to a single DB2 table Mapping of complex segment with embedded structures to multiple DB2 tables SQL-style WHERE clause for selection of propagated data

9 IBM Data Replication for IMS - the Selector 1 Updates to IMS segments logged as changed data records (9904) on IMS log. Log name registered in RECON (DBRC). IMS LOG IMS LOG IMS LOG IMS LOG RECON SCF 2 3 Selector reads Selector Control File, determines valid segment names and start and stop time for selection. Selector locates valid IMS logs via DBRC RECON. (CDCDSs if available) ULR Selector 4 Selector reads log datasets and writes only committed changed data records to PRDS in source commit order. PRDS 5 Uncommitted units of work written to ULR dataset.(transaction integrity) The Selector runs on the same source system as the IMS database, and can be set up to be run automatically at each IMS log archive

10 IBM Data Replication for IMS - the Receiver PRDS DIRECTORY TABLES 1 Receiver program reads directory table to determine next PRDS to process RECEIVER (RUP) 2 3 Receiver decompresses the changed data records for each segment in the PRDS. Receiver calls RUP program. RUP updates the target tables via static SQL calls for each PR. 4 Receiver continues sequentially processing PRDSs until stop criteria met. DB2 TARGET TABLES Stop Criteria for the DPropNR Receiver can include a special "stop processing" timestamp marker that is created by the user through a DPropNR application interface

11 About the IMS Replication Data transformation capabilities include: hierachical input to relational output at the field level, by the DPropNR code at the field or segment level by exit code sample exits provided in assembler, PLI, COBOL, and C user can provide by DB2 trigger on the target table through further processing during DPropR Apply

12 IBM Relational Replication Captures base table changes from log Capture runs locally to the source Maintains transaction consistency Automatically maintains staging tables SOURCE CONTROL TARGET APPLY SOURCE UNIT OF WORK CHANGE DATA TARGET TARGET TARGET Log Full Row Logging CONTROL ADMINISTRATION CAPTURE Base Tables Column Selection After Image or Before & After Image

13 IBM Relational Replication - Capture SOURCE BASETABLE Log Buffer Active Log 1 Source table changes are available first in the log buffers, then active logs, and then archive logs CAPTURE Archive Log 2 Capture uses IFI to read these log records. A buffer read is clearly the fastest! 3 Capture inserts changed data into the staging tables. UOW BASE TABLE CD TABLE 4 Capture deletes data from the staging tables after it has been applied. Typically the Capture program is always active when the DB2 database is active

14 IBM Relational Replication - Apply SOURCE UOW TABLE BASE CD TABLE 1 Data to be retrieved is joined and ordered at the source site Spill File TARGET 2 Data is then transported to the target into the spill file APPLY COPY BASE 3 Records are read from the spill file or from memory and SQL statements issued COPY BASE COPY BASE Apply can run on a timer scheduled basis or driven by an event Events can specify a timestamp as the end point of the source data to be processed

15 About the DB2 Replication Data transformational capabilities include: Horizontal and vertical subsetting available with both Capture and Apply processes Any valid SQL expression can be used in the fetch of the source data SQL calls and/or stored procedures can be called by Apply either before or after applying data Row level triggers can be used on both CD tables and target tables to further enhance data Views can be used on source or target tables to join, union, or translate data fields Apply target types include copy, history, and aggregates Changed data can be summarized automatically by defining an AST against the target table

16 DataJoiner Technical Summary DB2 for OS/390, Windows Transparency DB2 for VSE & VM or DB2 for OS/400 AIX,HP-UX, Solaris Single-DBMS Image Full SQL Global Optimization DB2 EEE DB2 for OS/2 DB2 for AIX DB2 for NT DB2 for HP-UX or DB2 for Solaris OS/2 DataJoiner VSAM IMS DB2 CS V2 Internet/Intranet Oracle MVS, AS/400,VSE AIX, Solaris, and NT Sybase Microsoft SQL Server DataJoiner access to other data sources can be combined to extend IBM Relational replication to include non-ibm data as sources and targets of replication Informix Teradata

17 IBM Relational Replication from Oracle DPropR Apply V7 DB2 V7 OS/390 DataJoiner 2.11 for Solaris Oracle 8.1 SQL*Net Replication triggers Replication Management Solaris Server DJRA on Windows Client LAN Network

18 About the Oracle Replication Data transformation capabilities include: Oracle to DB2 datatype conversion provided through DataJoiner data subsetting through user trigger modifications all subsetting and transformation capabilities of DPropR Apply Captured data is automatically pruned by triggers after data has been successfully processed by the Apply program

19 IBM Data Replication Components Full Refresh Source Data DB2 Family DataPropagator Relational Capture DB2 Update Staging Tables NotesPump Apply Notes Apply DB2 DataJoiner Multivendor Access Targets Multi-vendor Update DB2 Family Multi-vendor Capture Triggers DataPropagator NonRelational Full Refresh Capture IMS IMS, VSAM, Flat Files, Other DataRefresher Lotus Notes

20 About the CCD Table Format CCD - Consistent Changed Data Captured user data with 4 replication overhead columns timestamp sequencing information (tran commit order, order within tran) update type (I,U,D) Condensed or non-condensed append only (historical format) or update in place no deletes, deletes are logical - update type = 'D' Provides an excellent structure for an Operational Data Store all data timestamped, can partition data based on date allows downstream data stores to be updated with changed data only - no double capture, no full extracts data cleansing can be performed based on changed data, but with benefit of access to complete data

21 The CCD Table - Standard Replication Interface IMS DB2 CCD DB2 COPY COPY COPY DPropNR DPropR Apply DPropR Capture DB2 DB2 DB2 UOW BASE CD DPropR Apply CCD DB2 DPropR Apply DB2 COPY COPY COPY Oracle DB2 DB2 DPropR Triggers CCD DPropR Apply CCD DPropR Apply COPY COPY COPY

22 Pulling the Solution Together Processes that will run all of the time: DPropR Capture for OS/390 DPropNR Selector - automated by launching job through the IMS log archive process DPropR Capture triggers on Oracle database Processes that will be scheduled/managed: An application that creates an IMS "stop processing" marker, and posts event "end_of_period" timestamps for the DB2 and Oracle captured source data DPropNR Receiver DPropR Apply for OS/390 DPropR Apply for UDB Data Cleansing Programs Data Transformations

23 Desktop Manager Group View Desktop Help Main Desktop Manager Group View Desktop Help Main DB2 Warehouse Manager Data Sources DB2 FAMILY ORACLE SYBASE INFORMIX SQL SERVER Extract - Transform - Distribute Warehouse Agents NT, OS/2, AS/400, AIX, SUN, OS/390 Administrative Client Definition Management Operations Files Warehouse Server OTHER DataJoiner DataJoiner Classic Connect Metadata Information Catalog Data Access Tools IMS & VSAM DB2 Data Warehouses Transformers

24 Warehouse Center - Replication Management The Warehouse Manager can administer the subscription and manage the Apply of tables which have been previously registered for replication

25 DB2 Warehouse Manager - Automation Define workflow Scheduled or triggered Warehouse or User defined processes Dependency management Conditional processing Handle errors Automatic retry Error logging Monitor execution status Scheduled, in-progress, completed Log access notification Gather and report warehouse statistics Automatic statistics collection and reporting

26 DB2 Warehouse Manager - Statistical Transformers ANOVA Calculate statistics Calculate subtotals Chi-Square Correlation Moving average Regression

27 DB2 Warehouse Manager - Warehouse Transformers Clean Data Generate Key Table Generate Period Table Invert Data Pivot Data

28 IBM Response - A Complete Warehouse Solution Data Warehousing Solution DB2 for OS/390 V7 as Operational Data Store DB2 UDB EE V7 on AIX - Datamart and e-business servers Replication Solution Products DPropNR, DPropR, DataJoiner Data Management tools Control Center, DJRA, Warehouse Center DB2 Warehouse Manager Process Management Transformation Tools Metadata Access OLAP tools IBM Services - for an accelerated launch

29 Way Cool!! Airlines - Desired Configuration OLTP Systems Oracle 8.1 IMS V6 DB2 V5 OS/390 OS/390 Solaris Server Operational Data Store Data Warehouse e-business servers Aix Server Aix Server Aix Server Aix Server

30 Way Cool!! Airlines - Proposed Databases OLTP Systems Oracle 8.1 IMS V6 DB2 V5 OS/390 OS/390 Solaris Server Operational Data Store DB2 V7 Data Warehouse OS/390 e-business servers DB2 V7 EE DB2 V7 EE DB2 V7 EE DB2 V7 EE Aix Server Aix Server Aix Server Aix Server

31 Way Cool!! Airlines - Proposed DBs and Tools OLTP Systems IMS V6 DPropNR Selector OS/390 OS/390 DB2 V5 DPropR Capture Oracle 8.1 Solaris Server DataJoiner Operational Data Store DPropR Apply Data Warehouse Warehouse Mgr Agent OS/390 DPropNR Receiver DPropR Apply Warehouse Mgr Agent DB2 V7 e-business servers DPropR Apply DB2 V7 EE DB2 V7 EE DB2 V7 EE DB2 V7 EE Aix Server Aix Server Aix Server Aix Server

32 Web Information It all starts at... /db2/ /busn-intel/...for database info...for business intelligence info /dbtools/datarepl.html...for replication info