Luncheon Webinar Series January 13th, Free is Better Presented by Tony Curcio and Beate Porst Sponsored By:

Transcription

1 Luncheon Webinar Series January 13th, 2014 Free is Better Presented by Tony Curcio and Beate Porst Sponsored By: 1

2 Free is Better Questions and suggestions regarding presentation topics? - send to editor@dsxchange.com Downloading the presentation Replay will be available within one day with with details Pricing and configuration - send to editor@dsxchange.net Subject line : Pricing For those that stay through the entire presentation, we have a extra give away! Bonus Offer Free premium membership for your DataStage Management! Submit your management s address and we will offer him access on your behalf. Info@dsxchange.net subject line Managers special. Join us all at Linkedin 2

3 Free is Better Free data quality and governance components in DataStage 9.1.2

4 Session Abstract DataStage has been updated to include new features from our data quality portfolio that will help you jumpstart your quality led Data Governance objectives. Join this session to learn about the new "Exception Stage" and how it can be used to surface data issues to your data stewards through the Data Quality Console, a thin-client ui that is purpose-built for your data governance team... and, best of all, ALL included with your DataStage license. We'll also explore how QualityStage and Information Analyzer then expand these capabilities to round out the data quality lifecycle. FOR FREEEEEE?!?!?!?!

5 Agenda Included with DataStage Data Quality Console - Integrated Stewardship Web UI for Information Server Exception Stage - Allows DataStage and QualityStage data issues to be registered into the Data Quality Console Operational Quality - Monitors the health of the data integration environment

6 Data Quality Console - Introduction Web-based, unified environment to monitor & assess data quality Proactively increase awareness of data quality throughout the enterprise Increase business confidence when using enterprise data for critical decision making Expedites error correction by helping data governance teams focus on critical data issues InfoSphere Data Quality Console offers increased data quality awareness, insight, and management!

7 Data Quality Console - Overview Unified environment to proactively increase Data Quality awareness across multiple domains integration, validation, cleansing, etc Information Analyzer data validation and monitoring DataStage data integration & transformation QualityStage data cleansing, standardization and matching InfoSphere Data Quality Console

8 Roles Supporting Data Governance Community Business (Data) Steward Uses the Data Quality Console dashboard to understand quality trends Can browse and search common data quality event information Reviewer Owner of data quality events Investigate the record level to resolve the cause of the problem Can change the status of a data quality event Can change the owner of a data quality event Review Manager All Reviewer capabilities Plus the following: Assigns owners Change priorities Delete data quality events

9 Data Quality Console - Terminology Exception Descriptor Contains basic information about an exception such as: Date / Time of exception Priority, Status, Owner of that exception Originating System/Project Viewable/searchable by Business Steward, Reviewer Manager, and Reviewer Exception Details Contains any detail information passed from the exception producing system Validation exception as defined in an Information Analyzer Rule Source to target validation errors as defined in Information Analyzer Exception Manager Accessible only by Reviewer Manager and Reviewer roles

10 Data Quality Dashboard Home Page Ed logs into the console and switches to the Exceptions Tab to browse the exception summaries.

11 Exception Descriptor Search and Filtering To locate the exception summaries that are assigned to him, Ed opens the Owner facet and deselects the other owners

12 Exception Descriptor Search and Filtering Now, Ed is able to see all of the exceptions he owns. To focus on the most recent exceptions, Ed opens the Last Modified facet.

13 Exception Descriptor Search and Filtering Now Ed is able to see only the exceptions summaries he owns that were modified in the last day. He can view each exception summary in more detail by clicking on the exception summary name.

14 Exception Descriptor Search and Filtering Here Ed can view additional details about the exceptions, and can review more information about exactly what the exceptions were about.

15 Exception Drill Down He can also switch to the Exceptions tab to view the record-level exception instances. When finished, he returns to the exception summary list.

16 Exception Drill Down To pass this information to his team, Ed decides to export the list of exception summaries

17 Data Quality Console Summary Browser based user interface for simple comsumability Non-chargeable component Works with Information Analyzer, Information Analyzer Exception Manger, DataStage and QualityStage Primary information stored in the Information Server Repository with drill-down details available from product module sources. 201

18 Agenda Data Quality Console - Integrated Stewardship Web UI for Information Server Included with DataStage Exception Stage - Allows DataStage and QualityStage data issues to be registered into the Data Quality Console Operational Quality - Monitors the health of the data integration environment

19 Exception Stage - Introduction Collect exception data from any data integration or quality process and monitor over time Promote consistency in the way data stewards and business analysts can investigate data issues. Insert good data quality controls and governance practices into each project. Support clerical review for one-source and two-source variants to support business analyst review and tuning match algorithms Data steward dashboarding provides the same charting, searching, reporting, and monitoring as data rules, to facilitate integrated data remediation processes

20 Exception Stage - Types Exception stage Use to specify and gather information about exceptions that are generated by all stages except the One-source Match and Twosource Match stages. The Exceptions stage receives input that you identify as exception records. One-source Match Exception stage Use to specify and gather information about exceptions that are generated by One-source Match stages. The One-source Match Exceptions stage receives master record data and clerical record data from the One-source Match stage. Two-source Match Exception stage Use to specify and gather information about exceptions that are generated by Two-source Match stages. The Two-source Match Exceptions stage receives clerical review records from the Two-source Match stage.

21 Sample data scenarios that create ETL exceptions Database related Primary or foreign key constraint violation Could not delete/update (key does not exist) Database trigger failure Reference lookup not found Transformation Related Unexpected null value Data type not appropriate Data value not in expected range Constraint test not successful Sequential File Short read Delimiter not found Data format/type not appropriate Do you have a common process for how these are handled and reported to the data steward/ governance organization?

22 Exception Descriptor Editor Four tabs Categories Implemented Data Resources Additional Details Output Columns

23 Exception Descriptor Editor - Categories All exceptions stages have categories At least one is required Able to use more than one Available in Data Quality Console and are then used as Search Criteria

24 Exception Descriptor Editor Implemented Data Resources Allow you to associate these exceptions to a specific data source (optional)

25 Exception Descriptor Editor Additional Details Allow you to associate these exceptions to a specific stages on the canvas (optional)

26 Exception Descriptor Editor Additional Details (continued) The Other Properties flexibility means you can add: A description of the rules used to generate the exception A Data Governance Policy or Rule name for Business Glossary Anywhere to search and much more!

27 Exception Descriptor Editor Output Columns Will show in Data Quality Console as Exception Details

28 How It All Shows Drill-down to detailed exception records In this example, records with missing first name or age or gender.

29 Agenda Data Quality Console - Integrated Stewardship Web UI for Information Server Exception Stage - Allows DataStage and QualityStage data issues to be registered into the Data Quality Console Requires Information Analyzer Operational Quality - Monitors the health of the data integration environment

30 Scenarios for Operational Quality Consider the following examples: An organization has established a strict Service Level Agreement between business and IT that requires that the "Marketing Data Mart" is fully loaded before 6 AM each morning. A data integration center of excellence has established a policy that in order to optimize their resource utilization, their ETL Architect will review any data integration process that takes more than one hour. An organization that is growing through acquisition may want to have early warning indicators whether their data volume growth being loaded into the warehouse is exceeding the planned 20% increase each quarter. 30

31 Operational Quality - Definition How do you ensure that the day to day management of any solution is meeting IT and business expectations? Operational Quality is the conformance of information integration jobs or processes to established and required Service Level Agreements (SLA s) for the Availability of information, the Production of information, the minimization of Latency in information delivery, and the expected Utilization of system resources to produce the information delivered.

32 Operational Quality - Categories AVAILABILITY PRODUCTION LATENCY UTILIZATION Processes or jobs that deliver new or updated information to the Line of Business (LOB) system must execute on schedule and execute cleanly (i.e. minimal processing or job errors vs. average). The focus is evaluation of the job execution status and log. Processes or jobs that deliver new or updated information to the LOB system should produce consistent levels of data based on the data consumed. The focus is evaluation of the data volume produced by the job. Processes or jobs that deliver new or updated information to the LOB system must execute within the target time window. The focus is evaluation of the timing of the job execution. Processes or jobs that deliver new or updated information to the LOB system must execute within prescribed resource utilization parameters. The focus is evaluation of the system resources used by a given job.

33 Sample Operational Quality Rules

34 Operational Quality Accelerator Pre-built set of rules to measure Operational Quality now available on developerworks. Accelerator and doc intended to Be run as-is with minimal configuration against the DataStage and QualityStage Operations Console Database (DSODB) Reduce the effort in identifying operational quality issues within the categories noted above Serve as models, templates, and examples for your own additional operational rule design

35 Tremendous business value in an integrated platform Agile Integration Wherever your integration resides, integrate it quickly and flexibly Business Driven Governance Make decisions with confidence using trusted data at the point of impact Sustainable Quality Ensure information accuracy and quickly adapt to strategic business changes 35

36 Thank you

37 Free is Better Questions and suggestions regarding presentation topics? - send to editor@dsxchange.com Downloading the presentation Replay will be available within one day with with details Pricing and configuration - send to editor@dsxchange.net Subject line : Pricing For those that stay through the entire presentation, we have a extra give away! Bonus Offer Free premium membership for your DataStage Management! Submit your management s address and we will offer him access on your behalf. Info@dsxchange.net subject line Managers special. Join us all at Linkedin 37