Luncheon Webinar Series June 3rd, Deep Dive MetaData Workbench Sponsored By:

Size: px

Start display at page:

Download "Luncheon Webinar Series June 3rd, Deep Dive MetaData Workbench Sponsored By:"

Norman Higgins
6 years ago
Views:

1 Luncheon Webinar Series June 3rd, 2010 Deep Dive MetaData Workbench Sponsored By: 1

2 Deep Dive MetaData Workbench Questions and suggestions regarding presentation topics? - send to editor@dsxchange.com Downloading the presentation Replay will be available within one day with with details Pricing and configuration - send to editor@dsxchange.net Bonus Offer Free premium membership for your DataStage Management! Submit your management s address and we will offer him access on your behalf. Info@dsxchange.net subject line Managers special. Join us all at Linkedin 2

3 Tips and Tricks for Managing, Administering Metadata Successfully TSB-3403 Marc Haber Functional Architect, Infosphere Metadata Tools

4 Disclaimer Copyright IBM Corporation All rights reserved. U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY. WHILE EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE INFORMATION CONTAINED IN THIS PRESENTATION, IT IS PROVIDED AS IS WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. IN ADDITION, THIS INFORMATION IS BASED ON IBM S CURRENT PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM WITHOUT NOTICE. IBM SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION. NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, NOR SHALL HAVE THE EFFECT OF, CREATING ANY WARRANTIES OR REPRESENTATIONS FROM IBM (OR ITS SUPPLIERS OR LICENSORS), OR ALTERING THE TERMS AND CONDITIONS OF ANY AGREEMENT OR LICENSE GOVERNING THE USE OF IBM PRODUCTS AND/OR SOFTWARE. IBM, the IBM logo, ibm.com, Infosphere, and are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol ( or ), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at Copyright and trademark information at

5 Agenda Introduction InfoSphere Information Server InfoSphere Foundation Tools Metadata Primer Getting Started Goals Architecture Administration Tasks Product Demonstration Import, Manage and Deliver Summary and Conclusion

6 Introduction

7 InfoSphere Vision An Industry Unique Information Platform Simplify delivery of Trusted Information Accelerate Client Value Promote Collaboration Mitigate Risk Modular, yet Integrated Scalable Project to Enterprise 7

8 InfoSphere Information Server IBM InfoSphere Information Server Unified Deployment Discover, model, define, and govern information structure and content Standardize, merge, and correct information Combine and restructure information for new uses Synchronize, virtualize and move information for in-line delivery Unified Metadata Management

InfoSphere Foundation Tools Business Glossary Manage

Quality Metadata FastTrack Capture Design

Data Architect Design Enterprise Models Metadata

9 InfoSphere Foundation Tools Business Glossary Manage Business Terms Information Analyzer Assess Data Quality Metadata FastTrack Capture Design Specifications Discovery Understand Data Relationships Data Architect Design Enterprise Models Metadata Workbench Monitor Data Flows IBM Industry Models Leverage Industry Best Practices

InfoSphere Foundation Tools Portfolio Enterprise Projects Discover and understand the data across heterogeneous systems Design trusted information structures for business

Master Data Management InfoSphere Foundation Tools Data Warehousing Manage Business Terms Business Glossary Discover Discover Data Relationships New Discovery Design

10 InfoSphere Foundation Tools Portfolio Enterprise Projects Discover and understand the data across heterogeneous systems Design trusted information structures for business optimization Govern that information over time Test Data Generation Application Retirement & Consolidation Data Archival Data De-identification Data Quality Data Integration Master Data Management InfoSphere Foundation Tools Data Warehousing Manage Business Terms Business Glossary Discover Discover Data Relationships New Discovery Design Enterprise Models Data Architect Design Capture Design Specifications FastTrack Assess, Monitor, Manage Data Quality Information Analyzer Govern Monitor Data Flows Metadata Workbench 10

11 Infosphere Metadata Workbench Governance Asset catalog and metadata reporting for Data Governance initiatives and requirements Compliance Analysis Reporting for compliance measures in ensuring data quality and trust of Data Sources Standards Data Flow reports are requirements of Sarbanes Oxley, Basel II and other regulatory standards Change Understanding and reacting to the impact of change of Data Sources and structures

12 Metadata Primer Literally, data about data helps to describe a company s information from business, technical, and operational perspectives Practically, information that is important and critical, information that is difficult to grasp or fully understand, information that is continually emerging and processed

13 Metadata Primer standard definition Business Metadata Audience: Business users Purpose: Business rules, definitions, terminology, glossaries, algorithms and lineage using business language Technical Metadata Audience: Specific Tool Users BI, Data Integration, Profiling, Modeling Purpose: Defines source and target systems, table and field/attribute structures, derivations and dependencies Operational Metadata Audience: Operations, Management Purpose: Information about application runs: frequency, record counts, component by component analysis, other statistics

14 Metadata Primer user definition Meaning Understand the true meaning of a concept, what business process or entity does it represent, what business rules govern it, what specifications define it, what concepts are related Size and Construct Understand the length, type and structure of a concept Metrics Understand the cardinality, range, valid values, frequency of a concept Usage Trace the data flow through systems and applications, understand what processes and logic is involved in moving, transforming or otherwise aggregating data

15 Metadata Business Drivers Governance and Compliance Regulations are increasing How do organizations comply and meet documentation requirements? How can organizations ensure accountability and responsibility? Business Competition continues to grow How do organizations individualize their customer experience? How can organizations get access to information to make correct decisions? Costs and system complexities are expanding How can organizations drive optimization with integration? How do organizations manage complex software environments?

Metadata Primer Design Metadata Job Design Analysis: Analysis is defined as the projected flow of information, across different DataStage Jobs where the target and source Stages share a common source.

16 Metadata Primer Design Metadata Job Design Analysis: Analysis is defined as the projected flow of information, across different DataStage Jobs where the target and source Stages share a common source. Such information is necessary to determine the Impact of Change or Data Flow Analysis Reports delivered by the Infosphere Metadata Workbench. Linkage of Jobs via their common Stage Types and properties. Requires Automated Linkage service to be invoked Does not require user to load or use Physical Schema s or Files

Metadata Primer Operational Metadata Job Operational Analysis: Analysis is defined as the actual flow of information, from a Source data item through a set of actions defined within a DataStage Job

17 Metadata Primer Operational Metadata Job Operational Analysis: Analysis is defined as the actual flow of information, from a Source data item through a set of actions defined within a DataStage Job and written to a Target data item, based upon the Operational Job Run logs of the Job. Form a complete ETL Data Flow diagram, analyzing the sources of information, Job Run statistics and Transformation logic. Linkage of Jobs via their Job Run Operational Logs Requires import of Operational Metadata Requires Automated Linkage service to be invoked

InfoSphere Metadata Workbench Exploration and Analysis of Information Assets Features Explore, analyze and manage assets Data Lineage and Impact Analysis Extended visibility to enterprise integration

18 InfoSphere Metadata Workbench Exploration and Analysis of Information Assets Features Explore, analyze and manage assets Data Lineage and Impact Analysis Extended visibility to enterprise integration flows outside of Information Server Full searching and querying across information Assets Benefits Mitigate risk for change management Support compliance and governance initiatives Comprehensive understanding of data lineage for trusted information Project Managers & DBAs IT Developers Administrators

View context-specific details including stewards, term,

19 Data Lineage View end-to-end lineage including design metadata, operational metadata, user-defined metadata View context-specific details including stewards, term, description, Job image, Job operational metadata details, etc.

20 Business Lineage Business oriented view of Data Lineage Analysis report Business Lineage is configured within the Metadata Workbench, explicitly including only key Data Assets

21 Catalog and Display Data Catalog browse data structures, including Database, Data File, BI Report and Job assets Asset Details display asset information, including relationships and usage details

Asset Display Information Asset Information display

relationships Asset Usage understand ETL Jobs or

meaning, Data Steward, Mapping Specification

22 Asset Display Information Asset Information display base information, including description, container and relationships Asset Usage understand ETL Jobs or Mapping consumption, Business Glossary defined meaning, Data Steward, Mapping Specification requirement from FastTrack or Analysis Profiling data from Information Analyzer

23 Search and Query Homepage quickly search, display or query Information Assets Query Results formatted as a spreadsheet, for easier understanding and readability

24 Query Result Information Results Formatted as a spreadsheet, for easier understanding and readability Grouped according to Type Ability to save as Spreadsheet or Text File

25 Query Construction Create specific ad-hoc Reports Select Information Asset properties and Relationships or their propertiers Add specified conditioning filters Publish Queries for all users

26 Getting Started

27 Design Specification Design Document Abstract definition and specification which govern the flow of information from Source System for Reporting, OLAP and Mining deliverables. Governance and Auditing requirements dictate the need for Data Lineage reporting analysis.

28 Identify and Plan the Tasks System Application 2. Data File 3. Database Warehouse & Mart 4. BI Reports 5. DataStage Jobs 6. Data Scripts 7. Data Flow Analysis 4

29 Goals Data Lineage Ability to view Data Flow, validate Systems of Record, validate Business Logic Data Reporting Ensure compliance and data re-use, understand data consumption Data Terminology Ensure standardized language, descriptions and methodology Data Consistency Ensure proper Data Formatting, Data Type and Value Range

30 Metadata Preperation Import metadata about Database Tables and Files that are used in Job Design and Production Import metadata about BI Reports used to publish information Define and import Extended Data Sources and external Data Mappings for a complete end-to-end lineage flow Publish shared metadata as necessary Generate and import operational metadata from job runs Invoke Metadata Workbench administrative services Did you know? Design metadata for DataStage and QualityStage jobs is automatically stored in the metadata repository as well as metadata from all other suite tools.

31 Data Lineage and Impact Analysis

32 Data Reporting and Querying

33 Metadata Workbench Architecture AUTHOR AND LINK TO IT ASSETS BUSINESS GLOSSARY FAST TRACK METADATA WORKBENCH INFORMATION ANALYZER INFOSPHERE DATA ARCHITECT MANAGE CONTENT Metadata Workbench Data Structure METADATA SERVER ETL Design Business Technical Operational Lineage IMPORT/EXPORT MANAGER OR DATASTAGE CONNECTORS Querying ETL Operational IT ASSETS BI Structure BI REPORTS, PHYSICAL SCHEMAS, DS/QS JOBS Understanding

Infosphere Import Export Manager Features Import capabilities for 3 rd party BI tools (Cognos, Business Objects, MicroStrategy), data modeling tools (ERwin, RDA) and databases (ODBC connections to

34 Infosphere Import Export Manager Features Import capabilities for 3 rd party BI tools (Cognos, Business Objects, MicroStrategy), data modeling tools (ERwin, RDA) and databases (ODBC connections to all major RDBMS) Metadata Bridges interchange metadata with each specific application a consist of a model, a decoder, and an encoder which require no coding. Support a variety of import formats including XMI, XML, UML, CWM and CSV metadata exchange formats IT Developers IT Administrators Benefits Visibility of data modeling to ETL to report layer minimizes risks of overlooking critical dependencies Leverage common metadata exchange environment for application development consistency

Infosphere Import Extended Data Source Data

35 Infosphere Import Extended Data Source Data Source import and maintain application, procedure or file definitions from spreadsheets IT Developers IT Administrators Data Flow import and maintain source to target mappings, their business logic and function from spreadsheets

Infosphere Import Extended Data Mapping Data Flow Mapping document and

Developers IT Administrators Custom Attributes extend the properties of a

data, specification or organizational data Create or Import create

36 Infosphere Import Extended Data Mapping Data Flow Mapping document and express the transformation or business logic between source and target IT Developers IT Administrators Custom Attributes extend the properties of a mapping to record specific and proprietary information, including runtime data, specification or organizational data Create or Import create Extended Data Flow Mapping documents within the Metadata Workbench or import from a file

Infosphere Data Lineage Administration Metadata Administrators Ability to include or exclude Projects Intelligent metadata linking Ability to schedule Analysis Services Ability to map Database

37 Infosphere Data Lineage Administration Metadata Administrators Ability to include or exclude Projects Intelligent metadata linking Ability to schedule Analysis Services Ability to map Database Aliases Enhanced and extended support for Stages Allows administrators to minimize time maintaining and managing metadata assets as well as reduce the numbers of errors introduced from manual reconciliation processes.

DataStage and QualityStage Development As a developer

from the Source to the Target of the Job.

38 DataStage and QualityStage Development As a developer creates the Job canvass, they are building a flow of data from the Source to the Target of the Job. That flow, connected with other Job flows, will translate into Data Lineage. The Metadata Workbench Linkage Services will infer a relationship between both DataStage Jobs, based upon a common Data Set.

39 DataStage and QualityStage Job Design Ensuring a proper Job Design, while maintaining standards for naming and data connectivity will ensure greater linkages between the Job Design and the imported Data Source. Database Connectors Job Parameters and Environment Variables Load Column information from Shared Table Supported DataStage Stage Types DataStage Common Connector Stages Build SQL vs. User Defined SQL

40 Infosphere Data Lineage Support The following DataStage and QualityStage stages are supported by the IBM Metadata Workbench analysis service in determining cross Job relationships based upon the values of the Stage properties. Other types of DataStage Stages may be manually associated to Database Tables or Data File Elements. DB2 Native DB2 UDB API (S, P) DB2/UDB Enterprise (P) DB2 UDB Load (S, P) Server Name Schema Name Table Name RDBMS Native Dynamic RDBMS (S, P) Server Name Schema Name Table Name MSOLE Native MS OLEDB (S) Server Name Schema Name Table Name MSSQL Native Oracle Native Sybase Native MS SQL Server Load (S) SQL Server Enterprise (P) Oracle 7 Load (S) Oracle Enterprise (P) Oracle OCI (S) Oracle OCI Load (S) Sybase BCP Load (S) Sybase Enterprise (P) Sybase IQ 12 Load (S) Sybase OC (S) Server Name Schema Name Table Name Server Name Schema Name Table Name Server Name Schema Name Table Name ODBC ODBC (S) ODBC Connector (P) ODBC Enterprise (P) Server Name Schema Name Table Name TeraData Teradata API (S, P) Teradata Connector (P) Teradata Enterprise (P) Teradata Export (M) Teradata Load (S, M) Teradata Multiload (S, P) Teradata Relational (M) Server Name Table Name Complex Flat File Complex Flat File (S, P, M) File Name (S) = Server Canvas (P) = Parallel Canvas (M) = Mainframe Canvas Other Flat File Delimited Flat File (M) Fixed-width Flat File (M) Multi-format Flat File (M) File Name Hash File Hashed File (S) File Name Sequential File Sequential File (S, P) File Name or Pattern

41 Product Demonstration

42 Summary and Conclusion

43 Summary Step 1: Understanding the objectives Step 2: Defining the Tasks Step 3: IBM Infosphere Delivering Lineage and Understanding

44 Thank You! Your Feedback is Important to Us 44

45 Don t Miss these Foundation Tools Sessions!! Wed - May 19 Future Directions in Integrated Data Quality USL :00 PM - 04:00 PM Introduction and Overview - InfoSphere Foundation Tools Featuring Business Partner: Accantec Information Solutions TSB :00 PM - 03:50 PM The Evolution of a Complex Data Warehouse with InfoSphere Foundation Tools Customer: Consip S.p.a TSB :15 PM - 06:05 PM Building Business-led Informational Solutions with Industry Models, InfoSphere Warehouse, Business Glossary and Cognos TSB :15 PM - 06:05 PM Thu May 20 Data Discovery & Mapping to Accelerate Information Centric Projects TSB :45 AM - 08:45 AM Using Information Analyzer for Data Quality Health Monitoring TSB :45 AM - 08:45 AM Reduce costs, speed collaboration, and access critical data w/ low impact using Foundation Tools HOL :30 AM - 01:30 PM Get the Most Out of Your Data Modeling & Metadata Customer: Danske Bank TSB :45 AM - 12:35 PM A Metadata Based Approach to Data Governance Customer: Deutsche Bank BLD :00 PM - 02:50 PM InfoSphere Foundation Tools Deep Dive & Roadmap TSB :00 PM - 02:50 PM Governing Your Information Supply Chain TSB :00 PM - 02:50 PM Do You Really Trust Your Information? See How You Can - Live Demos Included TSB :00 PM - 02:50 PM ** Visit Our Live Demos Every The Demo Room! ** Understand and Map Your Distributed Data Integrated Metadata for Enterprise Collaboration and Trust Customer Sessions, Presentations, Usability Sessions, Live demos, Hands-On Labs Fri - May 21 Tips & Tricks for Managing & Administering Successful Metadata TSB :45 AM-08:45 AM Succeed In Getting All Stakeholders Involved Using Business Glossary TSB :45 AM-08:45 AM Delivering Smart Analytics, ROI & Business Benefits through the InfoSphere Portfolio Customer: 3UK BLD :00 AM 9:50 AM Accelerate Master Data Design and Definition using InfoSphere Discovery TSB :00 AM 9:50 AM Industry Models for Basel II Compliance and Risk Management Customer: CitiGroup BLD :30 PM 01:30 PM Assess Information Quality and Health Proven Models that Accelerate Your Information Agenda

46 Contacts us for more information about IBM InfoSphere Metadata Workbench Marc Haber Functional Architect, Metadata Tools Infosphere Metadata Workbench product specialist Farnaz Erfan Metadata Product Marketing Manager

Luncheon Webinar Series January 13th, Free is Better Presented by Tony Curcio and Beate Porst Sponsored By:

Luncheon Webinar Series January 13th, 2014 Free is Better Presented by Tony Curcio and Beate Porst Sponsored By: 1 Free is Better Questions and suggestions regarding presentation topics? - send to editor@dsxchange.com