Best practices for building a Hadoop Data Lake Solution CHARLOTTE HADOOP USER GROUP

Transcription

1 Best practices for building a Hadoop Data Lake Solution CHARLOTTE HADOOP USER GROUP

2 LANDING STAGING DW Let s start with something basic Is Data Lake a new concept? What is the closest we can relate it to as compared to the old world? SOURCE SYSTEMS ETL The evergreen left-to-right data flow for a typical data warehouse

3 Characteristics of the Landing Zone Landing Zone Closely resembles source definitions Truncate/Load Model predefined Mainly houses Structured data No user access or Analytics Typically, Landing zone is a bunch of tables which mimic source definitions Data in the Landing zone is usually not persisted Tables in Landing Zone usually are modeled and managed Landing zone is never a good option for any type of analytics No users have typically access to the Landing Zone Mainly suited for Structured data in Classic DW Architecture Slight variations Data Store (ODS or similar)

4 Need for a different type of Landing Zone Should be able to load any type of data Structured Classic world Semi-Structured , social media, text Unstructured media, logs, sensors Should not have to worry about building the model in advance Idea is to load and store the data instead of deciding what to load or what not to store Should not be constrained by the classic path of building the schema before loading Enable agility in Business Analytics Get the data to business for analysis faster Allow for deeper insights which were not possible before

5 Landing Zone Data Lake in perspective Governed Managed Limited Closed to outside world Selective data IT Driven Classic Landing Zone Modern Data Lake Not governed or partially Schema on read (dump) Not limited Open for data exploration Can host any type of data Analytics (business) driven What best practices should we follow to ensure the Data Lake Solution is: Address the needs Sustainable Maintainable Avoid turning the data lake into a swamp

6 Best Practices - Analysis Good stuff we can bring from the classic side: Data governance Metadata lineage Need to overcome these from the classic side: Development time for data ingestion pipelines Pre-defined Schema Constraint Limitation in data storage/loading/access methods Organic issues due to the charatectistics of data lake itself Lack of data discovery Data refinement concerns Data security concerns

7 Data is the new oil..but? Level of refinement dictates how good or effective the data can be Flexible data governance and metadata rules

8 Re-cap Best practices to build a data lake: Data Lake Solution should not create development bottlenecks for data ingestion pipelines allow any type of data to be loaded seamlessly and consistent manner should not require to build the target schemas before loading should not lock in to a technology pattern or behavior should allow for flexible data refinement policies allow for auto data discovery provide an agile development environment

9 Live Demo

10 How our solution address Big Data Integration challenges Sqoop Pig Hive HBase Map Reduce Spark Python Java Scala Spark Enables use of existing skillsets without the need to learn new/rapidly evolving technologies

11 Our Capabilities Data Ingestion Pipelines to Hadoop Fast, automated & manages target structures Data sources includes: DB2, SQL Server, IBM Netezza, Mainframe, Flat files, XML Bank systems involved: TLS, MIMS, MINS, E9, KQ, BCS, ALM Data Transformation & Preparation in Hadoop Quick, out of the box transformations for ready use Data platforms included: Hadoop BigSQL, DB2 EDW Bank systems involved: MIMS Staging on Hadoop, EDW Staging on DB2 Data Export to EDW Seamlessly export data from anywhere to DB2 EDW Data platforms included: Hadoop, BigSQL, DB2 Bank systems involved: EDW Audit, Balance & Control Accurate statistics and load control data for better insights into processes Control data includes Sequence Id, Audit Id, Dates, # of records extracted and loaded Integrated capabilities to provide operational dashboard using the statistics Metadata Lineage Forward and backward step by step and full lineage at Entity & Attribute level Diyotta integrates with Infosphere Information Governance Catalog (IGC) Inbuilt impact analysis for managing changes faster

12 Diyotta: for current and future requirements Data Ingestion Structured, semi-structured, unstructured Cloud or hybrid Batch driven or Services driven (REST API) Enterprise Data Lake Initiative Only solution in the market with out of the box Data Lake solution Enables data lake initiatives within days Ideal solution for either IT driven or business driven initiatives Future proofing towards technology evolution Change execution engine from MapReduce to Tez or Spark with one-click (no code change) Seamlessly port processing from one SQL to another SQL or Spark SQL Standardize code patterns and reusable components for maintainability Unique value proposition for Hadoop Metadata lineage and impact analysis within data lake to support data governance Unique extraction and load mechanism including Mainframe data processing in distributed manner Only solution with full pushdown on Hadoop no intermediate server

13 Diyotta Leading Modern Data Integration 1. Unified data integration approach on all Big Data platforms the same tool used for data ingestion and data transformation on Hadoop and traditional data warehouse platforms. You don t need to buy separate plugins for managing different platforms 2. Light footprint Server software is very lightweight and easy to install on an edge node or name node. All client modules are browser based and no need to install anything on users desktop 3. Target awareness no need to have any structures on the target (Hive, Hbase, or any other traditional databases). Automatically generates optimized structures on target side. 4. User choice of execution engine on Hadoop Supports Hive with MapReduce, Tez or Spark