METADATA BASED DYNAMIC ETLS. April 2015

Transcription

1 METADATA BASED DYNAMIC ETLS April 2015

2 Background Reports Generator ETL Architecture Problem of the Unknowns Final Architecture Implementing using SSIS How to solve the problem? Lessons Learnt AGENDA Capturing the metadata Using the meta-data Generating ETLs dynamically Challenges Mitigating using BIML

3 Reports Generator for Clearing Connectivity Standards Is a product To simplify integration with data systems That accepts Clearing Data and Generates Clearing Connectivity Standard Reports Background Raw Data A... Raw Data B Raw Data X Raw Data Y Clearing Reports Generator Std. Report 1... Std. Report n COPYRIGHT 2015 SAPIENT CORPORATION CONFIDENTIAL

4 Reports Generator High Level Architecture Raw Data Sources Schema A Schema B Schema C Reports Generator Extract Transform Std. Reports Schema X Schema Y Schema Z Load Schema Schema Known known during during Design design

5 Reports Generator ETL SSIS implementation for 1 Report Raw Data Source - A simple source schema from which an ETL package extracts the data Transform - Transformation on the extracted data Standard Report - A destination schema to load the data

6 Building a Product - The problem of the unknowns ABC corporation Raw Data Source Schema A XYZ corporation Raw Data Source Schema B A new prospect Source Schema? Report Generator ETLs (Point to Point) Report Generator ETLs (Point to Point) Report Generator ETLs (????) Standard Report Schema X Standard Report Schema Y Destination Schema? Takes time to build

7 How to solve the problem? 1. Capture the required meta-data Data about data Source & Destination Schema Info Tables, columns, Data type, expressions, etc. 2. Use the metadata at run-time A Bespoke solution A Do-All ETL package Dynamic ETLs Other options

8 1. Capturing the meta-data Meta-data store Source/Destination connection information Schema Information Source Schema Destination Schema Transformation Information Expressions Aggregations

9 2. Use the meta-data at run-time Dynamic ETLs Meta-data Store ETL Generator Read Metadata Validate Metadata Generate ETL dynamically ETL (SSIS) Packages

10 Generating Dynamic ETLs in SSIS Challenges/Risks Each SSIS ETL Package is a highly complex XML may vary for every version of SSIS not easy to maintain

11 Mitigating the risks - Simpler language for ETL About 660 lines of XML This is the whole package! <- OR ->? The Business Intelligence Markup Language (BIML) An XML dialect for ETL Simple and easy to maintain Can target multiple SSIS versions Powerful i.e. has inline C# support Reusable using templates

12 More About BIML... Created by Varigence Varigence is mainly into BI development Supports multiple SSIS versions Well supported by multiple IDEs/tools Mist and Vivid More about BIML in More about varigence in

13 Tying it back - Dynamic ETLs using BIML Metadata Store ETL Generator Read Metadata Validate Metadata Load BIML templates Load common templates Transform template to BIML Generate ETL dynamically ETL Packages

14 Reports Generator Final Architecture Metadata Store ETL Generator Read Metadata Validate Metadata Load BIML templates Load common templates Transform template to BIML Generate ETL dynamically Raw Data A Raw Data B... Raw Data X Raw Data Y ETL Packages Clearing Reports Generator Std. Report 1... Std. Report n

15 Lessons learnt A star schema on the RHS introduces maximum complexity. Use a flat schema Tracing the input from the output is difficult ETL as a technology is not for row by row Will it work? processing Use the abstraction layer wisely Improperly used BIML templates properly leads to maintainability issues Keep things simple - don t write business logic in BIML Maintain version changes across deployments COPYRIGHT 2015 SAPIENT CORPORATION CONFIDENTIAL

16 Background Reports Generator ETL Architecture Problem of the Unknowns Final Architecture Implementing using SSIS How to solve the problem? Lessons Learnt RECAP Capturing the metadata Using the meta-data Generating ETLs dynamically Challenges Mitigating using BIML

17 QUESTIONS

18 THANK YOU

19 APPENDIX

20 Appendix Transformation service architecture diagram

21 Transformation service