OWB Data Quality Best Practices

Size: px

Start display at page:

Download "OWB Data Quality Best Practices"

Rosanna Spencer
5 years ago
Views:

1 1

2 <Insert Picture Here> OWB Data Quality Best Practices Jean-Pierre Dijcks December 2008

3 Agenda Building a data quality firewall The importance of data rules The difference between profiling and auditing 3

4 Basic system architecture Data Sources Siebel CRM Oracle EBS Staging Data Layer Performance data layer SAP/R3 PeopleSoft Operational data layer Other Sources Message Queues 4

PeopleSoft Operational data layer Other Sources Message Queues Data

5 Building a data quality firewall Data Audits Data Audits Data Sources Siebel CRM Oracle EBS Staging Data Layer Performance data layer SAP/R3 PeopleSoft Operational data layer Other Sources Message Queues Data Profiling Schema & Data Type Correction Stage 2 Data Correction Data Governance 5

6 Building a data quality firewall Data Sources Siebel CRM Oracle EBS Staging Data Layer Operational data layer Performance data layer SAP/R3 PeopleSoft Other Sources Message Queues Profile Workspace Move Sample Data to Profile Workspace 6

7 Schema and Data Type Correction Leverage data profiling for Generating the staging area tables Schema corrections Data Type corrections (enforce real data types) Oracle EBS Untangle for lookups or recoding Profile data Discuss with business users Schema & Data Type Correction Staging Data Layer 7

8 Anatomy of the operational data layer Operational data layer Goal: Create lowest grain data for reporting Create a schema to service all applications with correct data Act as source for performance layer Characteristics De-normalized but still close to 3- NF Relationships established and enforced Data corrected and de-duplicated Permanent data 8

9 Loading the operational layer Leverage in-database architecture Do all the hard work here! Load between schemas not databases Huge performance gains through OWB architecture Embed data quality into the loads Create a data quality fire wall Strictly enforce all required rules Document all erroneous data and correct if desired Do matching and merging to create uniqueness from many data flows Create master data records Re-code as necessary Re-key as necessary Keep cross references 9

10 Data Quality Fire Wall Cleanse Protect Report Cleanse: De-duplicate incoming data Fix data issues Name and address String comparisons Operational data layer Protect: Enforce referential integrity Enforce data rules Enforce data types and conversions Report Data issues Quality levels Quality trends 10

11 Feeding non-dw systems Always load from the operational layer Cleanse Protect Report Delivers flexibility and lowest grain to external systems Aggregate on the way out if required (not typical) Operational data layer Delivers clean data, with measured service levels for DQ 11

12 Data Quality The importance of data rules Fear your data 1) Profile Know your data Ignorance Data Rules 2) Generate Correction Mappings Data Auditors 3) Operate 4) Monitor Coherent Data Audit Results and trends Trust your data Information 5) Report 12

13 Data Quality Data Profiling Unique Capabilities Complete offering Two usage modes: Use to investigate unknown data Use to validate known business rules against real data 13

14 Data Profiling vs. Data Auditing Both serve the same purpose through different means Data Profiling Ad-hoc when required Discovery in search of unknowns Time consuming Resource intensive Data Auditing: Continuous processes Planned to be done repetitively Gathers information over time Small tasks 14

15 <Insert Picture Here> D E M O N S T R A T I O N 15

16 Performance Tips for Data Profiling Data Profiling is a highly processor and I/O intensive process Run large profiles (>10M Rows in a table) on multiprocessor machines Use parallel: OWB uses /*+ PARALLEL(<TBL>) */ hints in DP queries Default degree of parallelism is picked up from database Balance your configuration Stripe data across disks using ASM Make sure I/O and CPU ratios are remotely correct 16

17 Performance Tips for Data Profiling When loading the workspace you are moving lots of data => optimize this: Place the profile workspace in the same database as the source data Enable the source tables for parallel reads Consider moving the data with regular OWB maps first, or use Transportable Tablespaces or Data Pump Memory: SGA should be no less than 500MB, preferably be around 2-3G for most profiles Buffer cache hit ratio >95% Library cache hit ratio >99% 17

18 Further Reference Material Data Quality posts about: Using data rules for Referential Integrity Key Quality Indicators Match and Merge Demonstrations on OTN Data Profiling and Corrections Fuzzy match and merging Name and address cleansing Training Extending your Knowledge (data profiling handson) 18

19 New Features for DQ Beta Program for 11gR2 If you are interested in the beta please contact the OWB product management team: Michelle Bird Or directly go to: Make sure to mention Michelle as sponsor. 19

20 Questions 20

21 High-Level Data Integration Roadmap Natural Upgrade Path for Existing Solutions Unified Team Summer 2009 CY 2011 Unified Platform Spring 2009 CY 2010 OWB/ODI Investments are Fully Protected No Forced Migrations Natural Upgrade Path Unified Platform aims to be a Superset of Existing Products no regression 21

22 22

<Insert Picture Here> Exadata MAA Best Practices Series Session 1: E-Business Suite on Exadata

<Insert Picture Here> Exadata MAA Best Practices Series Session 1: E-Business Suite on Exadata Exadata MAA Best Practices Series Session 1: E-Business Suite on Exadata Richard Exley Ray Dutcher Richard Exley, Ray Dutcher Oracle Applications, Exadata and MAA Best Practices Exadata