Talend Spark Meetup. Edward Ost Talend

Transcription

1 Talend Spark Meetup Edward Ost 2016 Talend 1

2 Talend: A History of Innovation and Growth Data Preparation Data Integration Data Quality Master Data Management Application Integration Big Data Hadoop 2.0 Spark & Cloud Key Facts Founded in employees worldwide 7 countries customers 2M+ open source downloads (Revenue Growth) 2

3 Top Big Data Challenges Talend Directly Addresses these Challenges Source: Gartner 12 September G Survey Analysis: Big Data Adoption in 2013 Shows Substance Behind the Hype 3

4 Talend 6.0 Unleashing the Power of Spark with Real-time Big Data Integration Talend Real-time Big Data The first data integration platform on Spark Internet of Things Delivers an end-to-end integration platform for IoT Continuous Delivery Provides Continuous Delivery data integration with unmatched productivity New Insight Easily access master data from Big Data, Mobile, and Cloud Apps using MDM REST APIs Smarter, More Secure Data New data masking and semantic discovery capabilities 4

5 Hadoop Cloud Hadoop Distros NoSQL Talend Remains Ahead of the Curve for Big Data BigInsights X X * x 3.x 3.x x * Tech Preview Talend 5.6.x (Dec 2014) Talend 6 (Sept 2015) Talend 6.1 (Dec 2015) Talend 6.2 (Jun 2016) 5

6 The More Data, The Better Talend Performs 7.8X Faster Benchmark MCG Global Services 2X Faster 3.5X 3.8X Faster Faster 5.4X Faster 7.8X Faster Number of Records Processed (in Millions) 6

7 Easily Convert MapReduce to Spark MapReduce Performance Spark Performance One Click (runs on disk) 5X (runs in-memory & Faster on disk) 7

8 Information Supply Chain Drivers Business Drivers Technical Concerns Evolving business network Data Broker ecosystem Transform Data into Information Onboarding data sources rapidly Accelerate insight Decouple source systems Increase agility Reduce process latency Avoid re-engineering At scale 8

9 DataVault Step 1: Establish the Business Keys, Hubs Step 2: Establish the relationships between the Business Keys, Links Step 3: Establish description around the Business Keys, Satellites Step 4: Add Standalone components like Calendars and code/descriptions for decoding in Data Marts Step 5: Tune for query optimization, add performance tables such as Bridge tables and Point-In-Time structures 9

10 Simple Data Vault Design Flow - Relational ACCOUNTS Account_ID (Pkey) Company_NM Address_LN1 Address_LN2 City State ZipCode Status_CODE Is_AUTHORIZED Is_LOCKED Created_DT Modified_DT USERS User_ID (Pkey) Account_ID (Fkey) First_NM Last_NM Mobile_PH Gender Status_CODE Is_ACTIVE Created_DT Modified_DT Identify Business Keys Identify Attributes Establish Linkages Control Lineage Control History De-Normalize 10

11 Simple Data Vault Design Flow Big Data ACCOUNTS Account_ID (Pkey) Company_NM Address_LN1 Address_LN2 City State ZipCode Status_CODE Is_AUTHORIZED Is_LOCKED Created_DT Modified_DT USERS User_ID (Pkey) Account_ID (Fkey) First_NM Last_NM Mobile_PH Gender Status_CODE Is_ACTIVE Created_DT Modified_DT Identify Business Keys Identify Attributes Establish Linkages Control Lineage Control History Create PIT & BRIDGE records De-Normalize 11

12 Spark and Data Vault Design Notes Focus on business keys and simplicity in source extracts Autonomous extracts enable parallel processing Capture and preserve auditable data in raw data vault Defer more complex business rules to the business vault Consider point-in-time tables for operational data vault 12

13 Basic Ingest 13

14 Data Vault Relational Model Extract data write to DV ready CSV files Push to S3/RDS Use ELT to De-Normalize into Columnar DataMart 14

15 Data Vault Big Data Analytics Sqoop data directly into S3/DV (Redshift) Use ELT to De-Normalize into Columnar DataMart 15

16 Data Vault with Spark Big Data Real Time Sqoop data directly into S3/DV (Hive) Transform to Data Vault with Spark Batch Operational Data Vault with Spark Streaming ELT to De-Normalize into Columnar DataMart 16

17 From Data to Information OLTP Systems of Engagement Data Warehouse Analytics BI Supply Chain Collaboration Self-Service On-Demand 17

18 Lambda Architecture Batch layer Load Analytics Reporting Data Discovery ERP NoSQL Systems of records DBMS Extract Transform Learn Data Mining MDD/OLAP Streaming layer Update Dashboarding IOT App. Events Web Logs Ingest Transform Act API Applications 18

19 Talend Big Data Resources Discover the Talend Big Data Jumpstart Sandbox Starting the Talend Big Data Sandbox Big Data Sandbox Forum Get It Right, in Real Time with SPARK Using AWS EMR, Redshift, and Spark to Power Your Analytics TalendForge Big Data Forum Data Vault Basics Data Vault Series Agile Modeling not an Option Anymore 19

20 Questions Edward Ost Channels Technical Director