Microsoft Developer Day

Transcription

1 Microsoft Developer Day

2 Pradeep Menon Microsoft Developer Day Solutions Architect

3 Agenda Microsoft Developer Day

4 Traditional Business Intelligence Architecture Structured Sources Extract Transform Structurize Load Enterprise Warehouse marts Reports OLAP Key Notes: Philosophy: Understand data. Transform data. Load data. Analyze data. More time spent on ETL. Less time for spend on analysis. discovery is a challenge. Not all data is analyzed.

5 in the Era of Big 1 in 3 leaders don t trust their data 100 TB of data per US company quality costs $600B/year 200 billion photos 600 years of videos Veracity Variety Big Volume Velocity 6 Billion smart phones 1PB of created every year Upto 100 sensors in modern car 0 million tweets/day 20 million n/w connections by 2020 The definition of analyzable data has changed. Traditional sources of data is disrupted.

6 Processing, Storage and Cost Moore s Law Processing capability has increased 10,000 times since 2000 Cost of storage has reduced by 1000 times since 2000 Capability to analyze more data efficiently 1 TB blob storage = $24/month

7 Lake Concept Lake Warehouse/Mart

8 Conceptual Lake Architecture Discovery Exploratory Analysis Predictive Modeling Structured Sources EL Raw Store Batch- Processing Engine Processed Stores Reports Unstructured Sources EL Real-Time Processing Engine Cataloging and Curation Security and Governance

9 Key Component Groups - Lambda Batch layer Structured Sources Unstructured Sources EL Lambda Architecture EL Raw Store Batch- Processing Engine Real-Time Processing Engine Speed layer Processed Stores Process: Batch Layer stores all the data in the rawest form possible, this set is the master data set. The master data set is immutable. follows a fact-based model, where each piece is atomic and timestamped. Streaming data follows both path. The speed layer allows random writes and the result are transient. Caters to both data on rest and data in motion

10 Key Component Groups- Analytical Sandboxes Discovery Exploratory Analysis Raw Store Analytical Sandboxes Predictive Modeling Batch- Processing Engine Processed Stores Process: Develop and test hypothesis. Explore data. Rapid prototyping. Discover data. Generate use-cases. Discover data. Extract value. Transform business

11 30 Key Component Groups Catalog and Governance Discovery Cataloging and Governance Exploratory Analysis Predictive Modeling How much does this painting cost? Structured Sources Unstructured Sources EL EL Raw Store Batch- Processing Engine Real-Time Processing Engine Processed Stores Cataloging and Curation Security and Governance Reports Catalog Information: The Old Guitarist Pablo Picasso 1903 Est: Over $100 million cataloging and curation is the key to extract value from data.

12 Key Component Groups Catalog Map Business Term Definition Logical Models Logical Entities More.. Business Term base Physical Model Tables Columns Primary keys Quality Score Schemas Lineage Information Class More.. Business Metadata Technical Metadata

13 EL Discovery EL Raw Store Exploratory Analysis Predictive Modeling Batch- Processing Engine Real-Time Processing Engine Processed Stores Cataloging and Curation Security and Governance Reports Difference between Lake and EDW Structured Sources Structured Sources Extract Transform Structurize Load Enterprise Warehouse marts Reports OLAP Unstructured Sources Lake Philosophy: Load first. Understand later. Retains all data i.e. stores data in its raw form. Supports all kinds of data Supports all kinds of users Adapts to change easily as requirements evolve Provides active cataloging of raw and transformed data Warehouse Philosophy: Understand first. Load later. Refined and transformed data structure for specific business requirement Optimized only for structured data stores Targeted to support operational users Relatively difficult to change as the data is highly structurized Provides limited metadata capture features

14 Azure Services for Lake Architecture Machine Learning Discovery Exploratory Analysis Predictive Modeling SSIS Factory EL Raw Store HDInsight Factory SSIS Batch- Processing Engine Processed Stores Reports Storage blob Lake Azure SQL database Stream Analytics Event Hubs Real-Time Processing Engine SQL Warehouse DocumentDB Cataloging and Curation Security and Governance Azure Access Active DirectoryControl

15 Key Takeaways 1. Lakes is a new paradigm shift for Big Architecture. 2. Lakes caters to all kinds of data, stores data in the raw form, caters to spectrum of users and enables faster insights. 3. Meticulous data cataloging and governance is key for successful data lake implementation. 4. Azure offers end-end solution for implementation of data lake architecture in an economical and scalable way.

16

17 Demo Architecture - Lambda Storage blob Event Hubs Stream Analytics

18 Demo Architecture Catalog Wine Quality Adventure Works Storage blob On-Prem SQL DB Cataloging and Curation

19 References Related references for you to expand your knowledge on the subject Azure Portal Azure Updates Microsoft Virtual Academy aka.ms/mva Developer Network msdn.microsoft.com/

20 Thank you Follow us online Microsoft Developer Day

21 Tell us what you think Help us shape future events by sharing your valuable feedback. Link - #MSDevDay