The Microsoft Big Data architecture approach

Size: px

Start display at page:

Download "The Microsoft Big Data architecture approach"

Morris Charles
5 years ago
Views:

February 2014 BASEL BERN BRUGG LAUSANNE ZUERICH UESSELORF

1 The Microsoft Big ata architecture approach Marc Schöni (Microsoft) Meinrad Weiss (Trivadis) 7. February 2014 BASEL BERN BRUGG LAUSANNE ZUERICH UESSELORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MUNICH STUTTGART VIENNA 1 The Microsoft Big ata architecture approach

2 Agenda 1. Introduction Big ata and Microsoft Products 2. Parallel ata Warehouse 3. Hadoop, Hadoop File System (HFS) and HInsight 4. Polybase: The bridge over the gap 2 The Microsoft Big ata architecture approach

3 Big ata: The three or four V s StreamInsight VOLUME (Size) VELOCITY (Speed) VARIETY (Structure) VALUE Hadoop/HInsight 3 The Microsoft Big ata architecture approach

4 Agenda 1. Introduction Big ata and Microsoft Products 2. Parallel ata Warehouse 3. Hadoop, Hadoop File System (HFS) and HInsight 4. Polybase: The bridge over the gap 4 The Microsoft Big ata architecture approach

Parallel ata Warehouse at a glance Shared-nothing parallel database system Massively parallel processing (MPP) A Control server that accepts user queries, generates a plan, and distributes operations

5 Parallel ata Warehouse at a glance Shared-nothing parallel database system Massively parallel processing (MPP) A Control server that accepts user queries, generates a plan, and distributes operations in parallel to compute nodes Multiple Compute servers running SQL Server elivered as an appliance Balanced and pre-configured hardware and software Scales from 2 to 56 Nodes Fastest Time to Market 5 The Microsoft Big ata architecture approach

6 Base Unit (1/4 Rack) All hosts and VM s run Windows Server 2012 Standard Control FA B A Host 0 Host 1 All Fabric and workload activity happens in Hyper-V virtual machines, with Fabric VM s sharing 1 server IB & Ethernet Compute 1 Compute 2 Host 2 Host 3 irect attached SAS Storage Spaces Failover is handled by Hyper-V PW Agent runs on all hosts and all VMs, collects appliance health data on fabric and workload Windows Storage Spaces handles mirroring and spares 6 The Microsoft Big ata architecture approach

7 SCALING FROM 2 TO 56 NOES Control FA B A Host 0 Appliance can grow with increasing workload Host 1 Compute 1 Compute 2 Host 2 Host 3 Storage Spaces Provisioning consists of 3 phases: Bare metal provisioning of new nodes IB & Ethernet 7 Compute 1 Compute 2 Host 2 Host 3 irect attached SAS Storage Spaces The Microsoft Big ata architecture approach Provisioning of workload VMs and hooking up to other workload VMs Redistribution of data

8 istribution and Replication of ata: Replicate Time im ate im I Calendar Year Calendar Qtr Calendar Mo Calendar ay Product im Prod im I Prod Category Prod Sub Cat Prod esc Smaller (<5GB ) imension Tables are Replicated on Every Compute Node SQL T S P M Sales Facts SQL T S P M Store im Store im I Store Name Store Mgr Store Size ate im I Store im I Prod im I Mktg Camp Id Qty Sold ollars Sold Mktg Campaign im Mktg Camp I Camp Name Camp Mgr Camp Start Camp End SQL SQL T S T S Result: Fact -imension Joins can be performed locally P M P M 8 The Microsoft Big ata architecture approach

9 Create Replicated Table CREATE TABLE improduct( ProductId BIGINT NOT NULL, escription VARCHAR(50), CategoryId INT NOT NULL, ListPrice ECIMAL(12,2)) WITH (ISTRIBUTION = REPLICATE); Creates tables on each of the individual compute nodes and assigns them to the REPLICATE file group. ata Compression is automatically turned on 9 The Microsoft Big ata architecture approach

istribution and Replication of ata: istribute Time im ate im I Calendar Year Calendar Qtr Calendar Mo Calendar ay Product im Prod im I Prod Category Prod

Store Mgr Store Size Sales Facts ate im I Store im I Prod im I Mktg Camp Id Qty Sold ollars Sold Mktg Campaign im Mktg Camp I Camp Name Camp Mgr Camp Start

10 istribution and Replication of ata: istribute Time im ate im I Calendar Year Calendar Qtr Calendar Mo Calendar ay Product im Prod im I Prod Category Prod Sub Cat Prod esc Larger (> 10 GB) Fact Table is Hash istributed Across All Compute Nodes SQL T S P M SF SF SF SF Store im Store im I Store Name Store Mgr Store Size Sales Facts ate im I Store im I Prod im I Mktg Camp Id Qty Sold ollars Sold Mktg Campaign im Mktg Camp I Camp Name Camp Mgr Camp Start Camp End SQL SQL SQL T S T S T S P M P M P M SF SF SF SF SF SF SF SF SF SF SF SF The Microsoft Big ata architecture approach

istribution on a PW CREATE TABLE mytable (column efs) WITH (ISTRIBUTION = HASH (id)); PW Node 1 Create Table <mytable GUI>_a Create Table <mytable GUI>_b Create Table <mytable GUI>_h 8 Tables per

11 istribution on a PW CREATE TABLE mytable (column efs) WITH (ISTRIBUTION = HASH (id)); PW Node 1 Create Table <mytable GUI>_a Create Table <mytable GUI>_b Create Table <mytable GUI>_h 8 Tables per Node PW Node 2 Create Table <mytable GUI>_a Create Table <mytable GUI>_b Create Table <mytable GUI>_h PW Node 11 PW Node 10 Create Table <mytable GUI>_a Create Table <mytable GUI>_b Create Table <mytable GUI>_h Final Result: 80 individual tables across a 10 node (1 data rack) appliance

12 Agenda 1. Introduction Big ata and Microsoft Products 2. Parallel ata Warehouse 3. Hadoop, Hadoop File System (HFS) and HInsight 4. Polybase: The bridge over the gap 12 The Microsoft Big ata architecture approach

13 Big, big data An other way of thinking 13 The Microsoft Big ata architecture approach

14 Hadoop and HInsight Collection of open sources projects in Apache for storing/processing big data (large, un/semi-structured data) Evolved over past 7+ years to power some of the largest data-driven sites/products The foundation/ kernel of HInsight 14 The Microsoft Big ata architecture approach

What is HInsight Enterprise grade big data platform Built on Hadoop in partnership with Hortonworks "Microsoft is far ahead of everyone else in terms of what they're contributing back

15 What is HInsight Enterprise grade big data platform Built on Hadoop in partnership with Hortonworks "Microsoft is far ahead of everyone else in terms of what they're contributing back to the community" Currently available as a service on Windows Azure Lets take a closer look Hortonworks Founder and Architect Arun Murthy 15 The Microsoft Big ata architecture approach

16 Introducing the Hadoop Ecosystem OBC istributed Processing (Map Reduce) World s ata (Azure ata Marketplace) istributed Storage (HFS) Windows Azure Storage 16 The Microsoft Big ata architecture approach

17 Windows Azure HInsight Service Query & Metadata: ata Movement: Workflow: Monitoring: Hadoop Filesystem Interface HFS Windows Azure Blob Storage 17 The Microsoft Big ata architecture approach

18 Windows Azure Storage Scalable, durable, and available Anywhere at anytime access Only pay for what the service uses Use from Windows Azure Compute Use from anywhere on the internet 18 The Microsoft Big ata architecture approach

19 atacenters and Regions West US East US Northern Europe Western Europe South Central US 19 The Microsoft Big ata architecture approach

20 Higher durability 3 local replicas in primary location Local replicas synchronously replicated Common failures (disk, node, rack) use local copies to recover Major disasters contact customer about potential data loss Reduced Price 23-34% based on how much you store Turn off Geo for your storage account in portal Non-critical data that can be recreated on major disasters Application manages its own replica Companies have limitations on geo locations Highest level of durability 3 local replicas each in primary and secondary locations Local replicas synchronously replicated Geo replica asynchronously replicated Common failures (disk, node, rack) use local copies to recover Major disasters use geo replicated copy (400+ miles apart) Price remains the same as before Enabled by default 20 The Microsoft Big ata architecture approach

21 Windows Azure HInsight Service 21 The Microsoft Big ata architecture approach

22 Consuming HInsight Result Sets Target estination Tool / Library Requires Active HInsight Cluster SQL Server, Sqoop (Hadoop ecosystem project) Yes Azure SQL B Excel Codename Power Query No Another Blob Storage Account SQL Server Analysis Services Existing BI Apps Azure Blob Storage REST APIs (Copy Blob, etc) Hive OBC river Hive OBC river (assumes app supports OBC connections to data sources) No Yes Yes 22 The Microsoft Big ata architecture approach

23 Agenda 1. Introduction Big ata 2. Parallel ata Warehouse 3. Hadoop, Hadoop File System (HFS) and HInsight 4. Polybase: The bridge over the gap 23 The Microsoft Big ata architecture approach

24 PW <-> Hadoop integration <-> Via Polybase PW introduces external tables to represent HFS data PW queries can now span HFS and PW data Social Apps Mobile Apps Sensor & RFI Web Apps HFS T SQL Enhanced PW query engine Known data structures 24 Semi-structured data The Microsoft Big ata architecture approach SELECT Username FROM ClickStream c,user u WHERE c.useri = u.i AN c.url= ; RBMS Structured data

25 And The Client Side 25 The Microsoft Big ata architecture approach

26 Big ata and Power BI: All together (Correlation Wind and Flight elay) 26

Microsoft Big ata Offerings: Cloud & On-Premises Hadoop

(minutes) Cost efficient Split of data and compute

SQL HFS (Hadoop) elivered as appliance Quick setup

27 Microsoft Big ata Offerings: Cloud & On-Premises Hadoop offering as PaaS in the cloud Extremely quick setup (minutes) Cost efficient Split of data and compute resources General big data on-premises platform atabase SQL HFS (Hadoop) elivered as appliance Quick setup Integrates all kind of data The Microsoft Big ata architecture approach

28 Questions and answers... Marc Schöni Technical Solution Professional Meinrad Weiss Principal Consultant BASEL BERN BRUGG LAUSANNE ZUERICH UESSELORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MUNICH STUTTGART VIENNA 28 The Microsoft Big ata architecture approach

Ian Choy. Technology Solutions Professional

Ian Choy. Technology Solutions Professional Ian Choy Technology Solutions Professional XML KPIs SQL Server 2000 Management Studio Mirroring SQL Server 2005 Compression Policy-Based Mgmt Programmability SQL Server 2008 PowerPivot SharePoint Integration