Luncheon Webinar Series June 3rd, 2010 Deep Dive MetaData Workbench Sponsored By: 1
Deep Dive MetaData Workbench Questions and suggestions regarding presentation topics? - send to editor@dsxchange.com Downloading the presentation http://www.dsxchange.net/metadataworkbench.html Replay will be available within one day with email with details Pricing and configuration - send to editor@dsxchange.net Bonus Offer Free premium membership for your DataStage Management! Submit your management s email address and we will offer him access on your behalf. Email Info@dsxchange.net subject line Managers special. Join us all at Linkedin http://tinyurl.com/dsxmembers 2
Tips and Tricks for Managing, Administering Metadata Successfully TSB-3403 Marc Haber Functional Architect, Infosphere Metadata Tools
Disclaimer Copyright IBM Corporation 2010. All rights reserved. U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY. WHILE EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE INFORMATION CONTAINED IN THIS PRESENTATION, IT IS PROVIDED AS IS WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. IN ADDITION, THIS INFORMATION IS BASED ON IBM S CURRENT PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM WITHOUT NOTICE. IBM SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION. NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, NOR SHALL HAVE THE EFFECT OF, CREATING ANY WARRANTIES OR REPRESENTATIONS FROM IBM (OR ITS SUPPLIERS OR LICENSORS), OR ALTERING THE TERMS AND CONDITIONS OF ANY AGREEMENT OR LICENSE GOVERNING THE USE OF IBM PRODUCTS AND/OR SOFTWARE. IBM, the IBM logo, ibm.com, Infosphere, and are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol ( or ), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at Copyright and trademark information at www.ibm.com/legal/copytrade.shtml
Agenda Introduction InfoSphere Information Server InfoSphere Foundation Tools Metadata Primer Getting Started Goals Architecture Administration Tasks Product Demonstration Import, Manage and Deliver Summary and Conclusion
Introduction
InfoSphere Vision An Industry Unique Information Platform Simplify delivery of Trusted Information Accelerate Client Value Promote Collaboration Mitigate Risk Modular, yet Integrated Scalable Project to Enterprise 7
InfoSphere Information Server IBM InfoSphere Information Server Unified Deployment Discover, model, define, and govern information structure and content Standardize, merge, and correct information Combine and restructure information for new uses Synchronize, virtualize and move information for in-line delivery Unified Metadata Management
InfoSphere Foundation Tools Business Glossary Manage Business Terms Information Analyzer Assess Data Quality Metadata FastTrack Capture Design Specifications Discovery Understand Data Relationships Data Architect Design Enterprise Models Metadata Workbench Monitor Data Flows IBM Industry Models Leverage Industry Best Practices
InfoSphere Foundation Tools Portfolio Enterprise Projects Discover and understand the data across heterogeneous systems Design trusted information structures for business optimization Govern that information over time Test Data Generation Application Retirement & Consolidation Data Archival Data De-identification Data Quality Data Integration Master Data Management InfoSphere Foundation Tools Data Warehousing Manage Business Terms Business Glossary Discover Discover Data Relationships New Discovery Design Enterprise Models Data Architect Design Capture Design Specifications FastTrack Assess, Monitor, Manage Data Quality Information Analyzer Govern Monitor Data Flows Metadata Workbench 10
Infosphere Metadata Workbench Governance Asset catalog and metadata reporting for Data Governance initiatives and requirements Compliance Analysis Reporting for compliance measures in ensuring data quality and trust of Data Sources Standards Data Flow reports are requirements of Sarbanes Oxley, Basel II and other regulatory standards Change Understanding and reacting to the impact of change of Data Sources and structures
Metadata Primer Literally, data about data helps to describe a company s information from business, technical, and operational perspectives Practically, information that is important and critical, information that is difficult to grasp or fully understand, information that is continually emerging and processed
Metadata Primer standard definition Business Metadata Audience: Business users Purpose: Business rules, definitions, terminology, glossaries, algorithms and lineage using business language Technical Metadata Audience: Specific Tool Users BI, Data Integration, Profiling, Modeling Purpose: Defines source and target systems, table and field/attribute structures, derivations and dependencies Operational Metadata Audience: Operations, Management Purpose: Information about application runs: frequency, record counts, component by component analysis, other statistics
Metadata Primer user definition Meaning Understand the true meaning of a concept, what business process or entity does it represent, what business rules govern it, what specifications define it, what concepts are related Size and Construct Understand the length, type and structure of a concept Metrics Understand the cardinality, range, valid values, frequency of a concept Usage Trace the data flow through systems and applications, understand what processes and logic is involved in moving, transforming or otherwise aggregating data
Metadata Business Drivers Governance and Compliance Regulations are increasing How do organizations comply and meet documentation requirements? How can organizations ensure accountability and responsibility? Business Competition continues to grow How do organizations individualize their customer experience? How can organizations get access to information to make correct decisions? Costs and system complexities are expanding How can organizations drive optimization with integration? How do organizations manage complex software environments?
Metadata Primer Design Metadata Job Design Analysis: Analysis is defined as the projected flow of information, across different DataStage Jobs where the target and source Stages share a common source. Such information is necessary to determine the Impact of Change or Data Flow Analysis Reports delivered by the Infosphere Metadata Workbench. Linkage of Jobs via their common Stage Types and properties. Requires Automated Linkage service to be invoked Does not require user to load or use Physical Schema s or Files
Metadata Primer Operational Metadata Job Operational Analysis: Analysis is defined as the actual flow of information, from a Source data item through a set of actions defined within a DataStage Job and written to a Target data item, based upon the Operational Job Run logs of the Job. Form a complete ETL Data Flow diagram, analyzing the sources of information, Job Run statistics and Transformation logic. Linkage of Jobs via their Job Run Operational Logs Requires import of Operational Metadata Requires Automated Linkage service to be invoked
InfoSphere Metadata Workbench Exploration and Analysis of Information Assets Features Explore, analyze and manage assets Data Lineage and Impact Analysis Extended visibility to enterprise integration flows outside of Information Server Full searching and querying across information Assets Benefits Mitigate risk for change management Support compliance and governance initiatives Comprehensive understanding of data lineage for trusted information Project Managers & DBAs IT Developers Administrators
Data Lineage View end-to-end lineage including design metadata, operational metadata, user-defined metadata View context-specific details including stewards, term, description, Job image, Job operational metadata details, etc.
Business Lineage Business oriented view of Data Lineage Analysis report Business Lineage is configured within the Metadata Workbench, explicitly including only key Data Assets
Catalog and Display Data Catalog browse data structures, including Database, Data File, BI Report and Job assets Asset Details display asset information, including relationships and usage details
Asset Display Information Asset Information display base information, including description, container and relationships Asset Usage understand ETL Jobs or Mapping consumption, Business Glossary defined meaning, Data Steward, Mapping Specification requirement from FastTrack or Analysis Profiling data from Information Analyzer
Search and Query Homepage quickly search, display or query Information Assets Query Results formatted as a spreadsheet, for easier understanding and readability
Query Result Information Results Formatted as a spreadsheet, for easier understanding and readability Grouped according to Type Ability to save as Spreadsheet or Text File
Query Construction Create specific ad-hoc Reports Select Information Asset properties and Relationships or their propertiers Add specified conditioning filters Publish Queries for all users
Getting Started
Design Specification Design Document Abstract definition and specification which govern the flow of information from Source System for Reporting, OLAP and Mining deliverables. Governance and Auditing requirements dictate the need for Data Lineage reporting analysis.
Identify and Plan the Tasks 1 2 3 3 6 5 5 1. System Application 2. Data File 3. Database Warehouse & Mart 4. BI Reports 5. DataStage Jobs 6. Data Scripts 7. Data Flow Analysis 4
Goals Data Lineage Ability to view Data Flow, validate Systems of Record, validate Business Logic Data Reporting Ensure compliance and data re-use, understand data consumption Data Terminology Ensure standardized language, descriptions and methodology Data Consistency Ensure proper Data Formatting, Data Type and Value Range
Metadata Preperation Import metadata about Database Tables and Files that are used in Job Design and Production Import metadata about BI Reports used to publish information Define and import Extended Data Sources and external Data Mappings for a complete end-to-end lineage flow Publish shared metadata as necessary Generate and import operational metadata from job runs Invoke Metadata Workbench administrative services Did you know? Design metadata for DataStage and QualityStage jobs is automatically stored in the metadata repository as well as metadata from all other suite tools.
Data Lineage and Impact Analysis
Data Reporting and Querying
Metadata Workbench Architecture AUTHOR AND LINK TO IT ASSETS BUSINESS GLOSSARY FAST TRACK METADATA WORKBENCH INFORMATION ANALYZER INFOSPHERE DATA ARCHITECT MANAGE CONTENT Metadata Workbench Data Structure METADATA SERVER ETL Design Business Technical Operational Lineage IMPORT/EXPORT MANAGER OR DATASTAGE CONNECTORS Querying ETL Operational IT ASSETS BI Structure BI REPORTS, PHYSICAL SCHEMAS, DS/QS JOBS Understanding
Infosphere Import Export Manager Features Import capabilities for 3 rd party BI tools (Cognos, Business Objects, MicroStrategy), data modeling tools (ERwin, RDA) and databases (ODBC connections to all major RDBMS) Metadata Bridges interchange metadata with each specific application a consist of a model, a decoder, and an encoder which require no coding. Support a variety of import formats including XMI, XML, UML, CWM and CSV metadata exchange formats IT Developers IT Administrators Benefits Visibility of data modeling to ETL to report layer minimizes risks of overlooking critical dependencies Leverage common metadata exchange environment for application development consistency
Infosphere Import Extended Data Source Data Source import and maintain application, procedure or file definitions from spreadsheets IT Developers IT Administrators Data Flow import and maintain source to target mappings, their business logic and function from spreadsheets
Infosphere Import Extended Data Mapping Data Flow Mapping document and express the transformation or business logic between source and target IT Developers IT Administrators Custom Attributes extend the properties of a mapping to record specific and proprietary information, including runtime data, specification or organizational data Create or Import create Extended Data Flow Mapping documents within the Metadata Workbench or import from a file
Infosphere Data Lineage Administration Metadata Administrators Ability to include or exclude Projects Intelligent metadata linking Ability to schedule Analysis Services Ability to map Database Aliases Enhanced and extended support for Stages Allows administrators to minimize time maintaining and managing metadata assets as well as reduce the numbers of errors introduced from manual reconciliation processes.
DataStage and QualityStage Development As a developer creates the Job canvass, they are building a flow of data from the Source to the Target of the Job. That flow, connected with other Job flows, will translate into Data Lineage. The Metadata Workbench Linkage Services will infer a relationship between both DataStage Jobs, based upon a common Data Set.
DataStage and QualityStage Job Design Ensuring a proper Job Design, while maintaining standards for naming and data connectivity will ensure greater linkages between the Job Design and the imported Data Source. Database Connectors Job Parameters and Environment Variables Load Column information from Shared Table Supported DataStage Stage Types DataStage Common Connector Stages Build SQL vs. User Defined SQL
Infosphere Data Lineage Support The following DataStage and QualityStage stages are supported by the IBM Metadata Workbench analysis service in determining cross Job relationships based upon the values of the Stage properties. Other types of DataStage Stages may be manually associated to Database Tables or Data File Elements. DB2 Native DB2 UDB API (S, P) DB2/UDB Enterprise (P) DB2 UDB Load (S, P) Server Name Schema Name Table Name RDBMS Native Dynamic RDBMS (S, P) Server Name Schema Name Table Name MSOLE Native MS OLEDB (S) Server Name Schema Name Table Name MSSQL Native Oracle Native Sybase Native MS SQL Server Load (S) SQL Server Enterprise (P) Oracle 7 Load (S) Oracle Enterprise (P) Oracle OCI (S) Oracle OCI Load (S) Sybase BCP Load (S) Sybase Enterprise (P) Sybase IQ 12 Load (S) Sybase OC (S) Server Name Schema Name Table Name Server Name Schema Name Table Name Server Name Schema Name Table Name ODBC ODBC (S) ODBC Connector (P) ODBC Enterprise (P) Server Name Schema Name Table Name TeraData Teradata API (S, P) Teradata Connector (P) Teradata Enterprise (P) Teradata Export (M) Teradata Load (S, M) Teradata Multiload (S, P) Teradata Relational (M) Server Name Table Name Complex Flat File Complex Flat File (S, P, M) File Name (S) = Server Canvas (P) = Parallel Canvas (M) = Mainframe Canvas Other Flat File Delimited Flat File (M) Fixed-width Flat File (M) Multi-format Flat File (M) File Name Hash File Hashed File (S) File Name Sequential File Sequential File (S, P) File Name or Pattern
Product Demonstration
Summary and Conclusion
Summary Step 1: Understanding the objectives Step 2: Defining the Tasks Step 3: IBM Infosphere Delivering Lineage and Understanding
Thank You! Your Feedback is Important to Us 44
Don t Miss these Foundation Tools Sessions!! Wed - May 19 Future Directions in Integrated Data Quality USL-3873 02:00 PM - 04:00 PM Introduction and Overview - InfoSphere Foundation Tools Featuring Business Partner: Accantec Information Solutions TSB-3392 03:00 PM - 03:50 PM The Evolution of a Complex Data Warehouse with InfoSphere Foundation Tools Customer: Consip S.p.a TSB-3333 05:15 PM - 06:05 PM Building Business-led Informational Solutions with Industry Models, InfoSphere Warehouse, Business Glossary and Cognos TSB-3593 05:15 PM - 06:05 PM Thu May 20 Data Discovery & Mapping to Accelerate Information Centric Projects TSB-3405 07:45 AM - 08:45 AM Using Information Analyzer for Data Quality Health Monitoring TSB-3410 07:45 AM - 08:45 AM Reduce costs, speed collaboration, and access critical data w/ low impact using Foundation Tools HOL-3845 10:30 AM - 01:30 PM Get the Most Out of Your Data Modeling & Metadata Customer: Danske Bank TSB-3496 11:45 AM - 12:35 PM A Metadata Based Approach to Data Governance Customer: Deutsche Bank BLD-3615 02:00 PM - 02:50 PM InfoSphere Foundation Tools Deep Dive & Roadmap TSB-3393 02:00 PM - 02:50 PM Governing Your Information Supply Chain TSB-3379 02:00 PM - 02:50 PM Do You Really Trust Your Information? See How You Can - Live Demos Included TSB-2902 02:00 PM - 02:50 PM ** Visit Our Live Demos Every Day @ The Demo Room! ** Understand and Map Your Distributed Data Integrated Metadata for Enterprise Collaboration and Trust Customer Sessions, Presentations, Usability Sessions, Live demos, Hands-On Labs Fri - May 21 Tips & Tricks for Managing & Administering Successful Metadata TSB-3403 07:45 AM-08:45 AM Succeed In Getting All Stakeholders Involved Using Business Glossary TSB-3414 07:45 AM-08:45 AM Delivering Smart Analytics, ROI & Business Benefits through the InfoSphere Portfolio Customer: 3UK BLD-3493 9:00 AM 9:50 AM Accelerate Master Data Design and Definition using InfoSphere Discovery TSB-3545 9:00 AM 9:50 AM Industry Models for Basel II Compliance and Risk Management Customer: CitiGroup BLD-3022 12:30 PM 01:30 PM Assess Information Quality and Health Proven Models that Accelerate Your Information Agenda
Contacts us for more information about IBM InfoSphere Metadata Workbench Marc Haber march@il.ibm.com Functional Architect, Metadata Tools Infosphere Metadata Workbench product specialist Farnaz Erfan erfan@us.ibm.com Metadata Product Marketing Manager