Microsoft Big Data and Hadoop

Transcription

1 Microsoft Big Data and Hadoop Lara Cindy

2 2 The world of data is changing

3 The 4Vs of Big Data 3

4 Common Big Data Patterns Digital Shoebox Information Production Information Factory Optimization Loop Retain all potentially valuable ambient data Analyze later Mine the digital shoebox for insight Feed downstream systems Produce consumable feeds Discover, enrich and publish Analyze ambient data Monitor system Optimize behavior 4

5 Scale Up! With the power of the Hubble telescope, we can take amazing pictures 45M light years away Analogous with scale up: non-commodity specialized equipment

6 Scale Out Commoditized Distribution but how about radio waves? Home: 5.2M participants, floating point operations 1, 769 teraflops 2 Analogous with commoditized distributed computing Distributed and calculated locally Engage with hundreds, thousands, + machines Many points of failure, auto-replication prevents problems

7 Hadoop: The most visible face of Big Data Enables analysis of semi and unstructured data distributed across a commodity cluster Infrastructure to automatically distribute and replicate data across multiple nodes and execute and track map reduce jobs across all of those nodes Based on Google s MapReduce paper and Google File system (GFS)

8 Zookeeper (Coordination) Hadoop Ecosystem Oozie (Workflow) Traditional BI Tools HBase / Cassandra (Columnar NoSQL Databases) Pig (Data Flow) Hive (Warehouse and Data Access) Karmasphere (Development Tool) Apache Mahout Flume Sqoop HBase (Column DB) MapReduce (Job Scheduling/Execution System) Hadoop = MapReduce + HDFS Avro (Serialization) HDFS (Hadoop Distributed File System)

9 Microsoft Hadoop Deployment Models Name Node Name Node Name Node On-Demand Azure Cluster On-demand elastic Hadoop cluster Bursty / batchy workloads Simplified management and deployment Complete elasticity Dedicated Azure Cluster Dedicated Hadoop cluster in the cloud Cloud workloads with more stringent storage / latency requirements Some elasticity (depending on HDFS storage and partitioning) Dedicated Windows Cluster Dedicated Hadoop cluster on Windows Server On premise workloads Management integration Elastic MapReduce Dedicated Cloud Cluster Dedicated On Premise Cluster

10 Demo As Featured on Slide 26 10

11 Comparing RDBMS and MapReduce Traditional RDBMS MapReduce Data Size Terabytes Petabytes Access Interactive and Batch Batch Updates Read / Write many times Write once, Read many times Structure Static Schema Dynamic Schema Integrity High (ACID) Low Scaling Nonlinear Linear DBA Ratio 1:40 1:3000 Reference: Tom White s Hadoop: The Definitive Guide

12 Hive Data warehouse system for Hadoop Batch analytics and Ad-hoc queries via HiveQL (translate into MapReduce) Easy data summarization Connect from PowerPivot, Excel, etc via Hive ODBC driver 14

13 JavaScript JavaScript console for Hadoop Interactive development environment with charting and graphing Write MapReduce jobs and Hive Queries in JavaScript

14 SQOOP Available today! Bi-directional data connector system for Hadoop and SQL Import SQL Server data to HDFS or tables in Hive Exporting files and Hive tables to SQL Server tables Executes MapReduce jobs to transfer data in parallel with fault tolerance 16

15 Pig Pig is a data-flow platform to transform and analyze large data sets on Hadoop Ease of programming through data flow sequences - Simpler to write & maintain Extensible through user defined functions and methods

16 Demo An Azure Elephant Never Forgets 18

17 Microsoft Big Data Solution External Data and Services MARKETPLACE

18 Where to learn more Hadoop: The Definitive Guide by Tom White Programming Pig by Alan Gates Hive: Sqoop: Pig Programming Pig by Alan Gates Javascript:

19

20

21

22

23