Big Data Analytics. Map-Reduce and Spark. Slides by: Joseph E. Gonzalez Guest Lecturer: Vikram Sreekanti

Transcription

1 Big Data Analytics Map-Reduce and Spark Slides by: Joseph E. Gonzalez Guest Lecturer: Vikram Sreekanti

2 From SQL to Big Data (with SQL) Ø A few weeks ago Ø Databases Ø (Relational) Database Management Systems Ø SQL: Structured Query Language Ø Today Ø More on databases and database design Ø Enterprise data management and the data lake Ø Introduction to distributed data storage and processing Ø Spark

3 Data in the Organization A little bit of buzzword bingo!

4 Inventory How we like to think of data in the organization

5 The reality Sales (Asia) Inventory Sales (US) Advertising

6 Sales (Asia) Sales (US) Inventory Advertising Operational Data Stores Ø Capture the now Ø Many different databases across an organization Ø Mission critical be careful! Ø Ø Serving live ongoing business operations Managing inventory Ø Different formats (e.g., currency) Ø Different schemas (acquisitions...) Ø Live systems often don t maintain history We would like a consolidated, clean, historical snapshot of the data.

7 Sales (Asia) Sales (US) ETL ETL Data Warehouse Collects and organizes historical data from multiple sources Inventory Advertising ETL ETL Data is periodically ETLed into the data warehouse: Ø Extracted from remote sources Ø Transformed to standard schemas Ø Loaded into the (typically) relational (SQL) data system

8 Extract à Transform à Load (ETL) Extract & Load: provides a snapshot of operational data Ø Historical snapshot Ø Data in a single system Ø Isolates analytics queries (e.g., Deep Learning) from business critical services (e.g., processing user purchase) Ø Easy! Transform: clean and prepare data for analytics in a unified representation Ø Difficult à often requires specialized code and tools Ø Different schemas, encodings, granularities

9 Sales (Asia) Sales (US) ETL ETL Data Warehouse Collects and organizes historical data from multiple sources Inventory Advertising ETL ETL How is data organized in the Data Warehouse?

10 Example Sales Data pname category price qty date day city state country Corn Food /30/16 Wed. Omaha NE USA Corn Food /31/16 Thu. Omaha NE USA Corn Food /1/16 Fri. Omaha NE USA Galaxy Phones /30/16 Wed. Omaha NE USA Ø Big table: many columns and rows Ø Ø Substantial Galaxy redundancy Phones 18 à expensive 20 3/31/16 to store Thu. Omaha NE USA and access Make Galaxymistakes Phones while updating /1/16 Fri. Omaha NE USA Ø Could we organize the data more efficiently? Galaxy Phones /30/16 Wed. Omaha NE USA Peanuts Food /31/16 Thu. Seoul Korea Galaxy Phones /1/16 Fri. Seoul Korea

11 Multidimensional Data Model Sales Fact Table pid timeid locid sales Locations locid city state country 1 Omaha Nebraska USA 2 Seoul Korea 5 Richmond Virginia USA Products pid pname category price 11 Corn Food Galaxy 1 Phones Peanuts Food 2 Time timeid Date Day 1 3/30/16 Wed. 2 3/31/16 Thu. 3 4/1/16 Fri. Dimension Tables Ø Multidimensional Cube of data Product Id Location Id Time Id

12 Multidimensional Data Model Sales Fact Table pid timeid locid sales Locations locid city state country 1 Omaha Nebraska USA 2 Seoul Korea 5 Richmond Virginia USA Products pid pname category price 11 Corn Food Galaxy 1 Phones Peanuts Food 2 Time timeid Date Day 1 3/30/16 Wed. 2 3/31/16 Thu. 3 4/1/16 Fri. Ø Ø Ø Ø Dimension Tables Fact Table Ø Minimizes redundant info Ø Reduces data errors Dimensions Ø Easy to manage and summarize Ø Rename: Galaxy1 à Phablet Normalized Representation How do we do analysis? Ø Joins!

13 The Star Schema Products Time pid pname category price timeid Date Day Sales Fact Table pid timeid locid sales ß This looks like a star Locations locid city state country

14 The Star Schema ß This looks like a star

15 The Star Schema ß This looks like a star

16 The Snowflake Schema ß This looks like a snowflake?

17 Which schema illustration would best organize this data? Data(calid, student_name, year, major, major_grade_req, asg1_name, asg1_pts, asg1_score, asg1_grader_name asg2_name, asg2_pts, asg2_score, asg2_grader_name avg_grade) Grades Students Graders Students Graders Majors Graders Grades Grades Majors (A) (B) (C) Assignments Assignments Students

18 Data(calid, student_name, year, major, major_grade_req, asg1_name, asg1_pts, asg1_score, asg1_grader_name asg2_name, asg2_pts, asg2_score, asg2_grader_name avg_grade) Grades(calid, asg_name, grader_id, score) Graders(grader_id, grader_name) Student(calid, name, year, major_name, avg_grade) Majors(major_name, grade_req) Assignments(asg_name, asg_pts) Grades Students Graders Students Graders Majors Graders Grades Grades Majors (A) (B) (C) Assignments Assignments Students

19 Online Analytics Processing (OLAP) Users interact with multidimensional data: Ø Constructing ad-hoc and often complex SQL queries Ø Using graphical tools that to construct queries Ø Sharing views that summarize data across important dimensions

20 Cross Tabulation (Pivot Tables) Item Color Quantity Desk Blue 2 Desk Red 3 Sofa Blue 4 Sofa Red 5 Color Item Desk Sofa Sum Blue Red Sum Ø Aggregate data across pairs of dimensions Ø Pivot Tables: graphical interface to select dimensions and aggregation function (e.g., SUM, MAX, MEAN) Ø GROUP BY queries Ø Related to contingency tables and marginalization in stats. Ø What about many dimensions?

21 Cube Operator Ø Generalizes crosstabulation to higher dimensions. ØIn SQL: SELECT Item, Color, SUM(Quantity) AS QtySum FROM Furniture GROUP BY CUBE (Item, Color); Item Color Quantity Desk Blue 2 Desk Red 3 Sofa Blue 4 Sofa Red 5 * Product Id Location Id 1 2 * Time Id Item Color QtySum Desk Blue 2 Desk Red 3 Desk * 5 Sofa Blue 4 Sofa Red 5 Sofa * 9 * * 14 * Blue 6 * Red 8 * What is here?

22 Ø Slicing: selecting a value for a dimension OLAP Queries Ø Dicing: selecting a range of values in multiple dimension

23 Ø Rollup: Aggregating along a dimension OLAP Queries Ø Drill-Down: de-aggregating along a dimension Day 3 Day Day Day 3 Day Day

24 Reporting and Business Intelligence (BI) Ø Use high-level tools to interact with their data: Ø Automatically generate SQL queries Ø Queries can get big! Ø Common!

25 Sales (Asia) Sales (US) ETL ETL Data Warehouse Collects and organizes historical data from multiple sources Inventory Advertising ETL ETL So far Ø Star Schemas Ø Data cubes Ø OLAP Queries

26 Sales (Asia) Sales (US) Inventory Advertising Text/Log Data ETL ETL ETL ETL ETL? Photos & Videos It is Terrible! Data Warehouse Collects and organizes historical data from multiple sources Ø How do we deal with semistructured and unstructured data? Ø Do we really want to force a schema on load?

27 Sales (Asia) Data Warehouse How do we cleane Sales (US) this TL and organize data? Inventory ETL Advertising ETL Collects and organizes historical data from multiple sources Depends on use ETL Text/Log Data ET? L Photos & Videos Ø How do we deal with semihow do we load and process this structured and unstructured data in a relational system? data? Depends on use Canto beforce difficult... Ø Do we really want a Requires thought... schema on load? It is Terrible!

28 Sales (Asia) Sales (US) Data Lake* ETL Store a copy of all the data Inventory ETL Ø in one place Advertising ETL Ø in its original natural form ETL Text/Log Data ET? L Photos & Videos It is Terrible! Enable data consumers to choose how to transform and use data. Ø Schema on Read What could go wrong? *Still being defined [Buzzword Disclaimer]

29 The Dark Side of Data Lakes Ø Cultural shift: Curate à Save Everything! Ø Noise begins to dominate signal Ø Limited data governance and planning Example: hdfs://important/joseph_big_file3.csv_with_json Ø What does it contain? Ø When and who created it? Ø No cleaning and verification à lots of dirty data Ø New tools are more complex and old tools no longer work Enter the data scientist

30 A Brighter Future for Data Lakes Enter the data scientist Ø Data scientists bring new skills Ø Distributed data processing and cleaning Ø Machine learning, computer vision, and statistical sampling Ø Technologies are improving Ø SQL over large files Ø Self describing file formats & catalog managers Ø Organizations are evolving Ø Tracking data usage and file permissions Ø New job title: data engineers

31 How do we store and compute on large unstructured datasets Ø Requirements: Ø Handle very large files spanning multiple computers Ø Use cheap commodity devices that fail frequently Ø Distributed data processing quickly and easily Ø Solutions: Ø Distributed file systems à spread data over multiple machines Ø Assume machine failure is common à redundancy Ø Distributed computing à load and process files on multiple machines concurrently Ø Assume machine failure is common à redundancy Ø Functional programming computational pattern à parallelism

32 Distributed File Systems Storing very large files

33 Fault Tolerant Distributed File Systems Big File How do we store and access very large files across cheap commodity devices? [Ghemawat et al., SOSP 03]

34 Fault Tolerant Distributed File Systems Big File Machine1 Machine 2 Machine 3 Machine 4 [Ghemawat et al., SOSP 03]

35 Fault Tolerant Distributed File Systems A B Big File C D Ø Split the file into smaller parts. Ø How? Ø Ideally at record boundaries Ø What if records are big? Machine1 Machine 2 Machine 3 Machine 4 [Ghemawat et al., SOSP 03]

36 Fault Tolerant Distributed File Systems B Big File C D Machine1 Machine 2 Machine 3 Machine 4 A A A [Ghemawat et al., SOSP 03]

37 Fault Tolerant Distributed File Systems Big File C D Machine1 Machine 2 Machine 3 Machine 4 B A B A A B [Ghemawat et al., SOSP 03]

38 Fault Tolerant Distributed File Systems Big File D Machine1 Machine 2 Machine 3 Machine 4 B C A B A C A B C [Ghemawat et al., SOSP 03]

39 Fault Tolerant Distributed File Systems Big File Machine1 Machine 2 Machine 3 Machine 4 B C A B A C A B D C D D [Ghemawat et al., SOSP 03]

40 Fault Tolerant Distributed File Systems Ø Split large files over multiple machines Ø Easily support massive files spanning machines Big File Ø Read parts of file in parallel Ø Fast reads of large files Ø Often built using cheap commodity storage devices Cheap commodity storage devices will fail! Machine1 B C D Machine 3 A C D Machine 2 A B C Machine 4 A B D

41 Fault Tolerant Distributed File Systems Failure Event Big File Machine1 Machine 2 Machine 3 Machine 4 B C A B A C A B D C D D [Ghemawat et al., SOSP 03]

42 Fault Tolerant Distributed File Systems Failure Event Big File Machine1 B C D Machine 2 Machine 3 Machine 4 A B A C A B C D D [Ghemawat et al., SOSP 03]

43 Fault Tolerant Distributed File Systems Failure Event Big File Machine 2 Machine 4 A B A B C D [Ghemawat et al., SOSP 03]

44 Fault Tolerant Distributed File Systems Failure Event A Big File C B D Machine 2 Machine 4 A B A B C D [Ghemawat et al., SOSP 03]

45 Map-Reduce Distributed Aggregation Computing are very large files

46 How would you compute the number of occurrences of each word in all the books using a team of people?

47 Simple Solution

48 Simple Solution 1) Divide Books Across Individuals

49 Simple Solution 1) Divide Books Across Individuals 2) Compute Counts Locally Word Count Apple 2 Bird 7 Word Count Apple 0 Bird 1

50 Simple Solution 1) Divide Books Across Individuals 2) Compute Counts Locally 3) Aggregate Tables Word Count Apple 2 Bird 7 Word Count Apple 2 Bird 8 Word Count Apple 0 Bird 1

51 The Map Reduce Abstraction Key Value Value Record Map Key Reduce Key Value Key Value Value Example: Word-Count Map(docRecord) { for (word in docrecord) { emit (word, 1) } Key Value } Reduce(word, counts) { emit (word, SUM(counts)) } Map: Deterministic Reduce: Commutative and Associative [Dean & Ghemawat, OSDI 04]

52 Key properties of Map And Reduce Ø Deterministic Map: allows for re-execution on failure Ø If some computation is lost we can always re-compute Ø Issues with samples? Ø Commutative Reduce: allows for re-order of operations Ø Reduce(A,B) = Reduce(B,A) Ø Example (addition): A + B = B + A Ø Is floating point math commutative? Ø Associative Reduce: allows for regrouping of operations Ø Reduce(Reduce(A,B), C) = Reduce(A, Reduce(B,C)) Ø Example (max): max(max(a,b), C) = max(a, max(b,c))

53 Executing Map Reduce Record Map Key Key Value Value

54 Machine1 B C D Executing Map Reduce Machine 2 A B C Machine 3 A C D Machine 4 A B D Map Distributing the Map Function

55 Machine1 B C D Machine 2 A B C Machine 3 A C D Machine 4 A B D Map Map Map Map Executing Map Reduce Distributing the Map Function

56 Machine1 B C D Machine 2 A B C Machine 3 A C D Map Map Map apple cat the big cat the cat dog Executing Map Reduce The map function applied to a local part of the big file. Run in Parallel. Machine 4 A B D Map big dog 1 1 Output is cached for fast recovery on node failure

57 Machine1 B C D Machine 2 A B C Map Map apple cat the big cat the Executing Map Reduce Machine 3 A C D Map cat dog 1 1 Key Value Value Reduce Key Value Machine 4 A B D Map big dog 1 1 Reduce function can be run on many machines

58 Machine1 B C D Machine 2 A B C Machine 3 A C D Map Map Map Run in Parallel apple 1 Reduce apple 1 the big Reduce the 3 cat Reduce cat 5 Reduce big 2 Machine 4 A B D Map dog 1 1 Reduce dog 2

59 Machine 2 A B C Machine 4 A B D Machine 3 A C D Machine1 B C D Map Map Map Map Reduce 1 big 1 1 apple Reduce 2 the 1 Reduce 3 1 Reduce 1 cat Reduce 1 1 dog 2 big 1 apple 3 the 5 cat 2 dog Output File

60 Machine1 B C D Machine 2 A B C Machine 3 A C D Map Map Map apple the cat big Reduce Reduce Reduce Reduce Output File apple the cat big dog If part of the file or any intermediate computation is lost we can simply recompute it without recomputing everything. Machine 4 A B D Map dog 1 1 Reduce

61 Interacting with Scale Map-Reduce

62 Interacting With the Data Good for smaller datasets Ø Faster more natural interaction Ø Lots of tools! Compute Locally = M r2data f (r) Request Data Sample Sample of Data Analysis Algorithm Computation Can we send the computation to the data? Yes!

63 Statistical Query Pattern Common Machine Learning Pattern Ø Computing aggregates of user defined functions Ø Data-Parallel computation = M r2data f (r) Iterative Algorithm Query:f Response: f : User defined function [UDF] M : User defined aggregate [UDA] D. Caragea et al., A Framework for Learning from Distributed Data Using Sufficient Statistics and Its Application to Learning Decision Trees. Int. J. Hybrid Intell. Syst. 2004

64 = M f (r) = M f (r) r2data r2data Query:f Response: M Query:f Response: Query:f = M r2data f (r) Iterative Algorithm Response:

65 Interacting With the Data = M r2data f (r) Good for smaller datasets Ø Faster more natural interaction Ø Lots of tools! Compute Locally = M r2data f (r) Request Data Sample Sample of Data Algorithm Query:f Response: Algorithm Good for bigger datasets and compute intensive tasks Cluster Compute

66 Map Reduce Technologies

67 Hadoop Ø First open-source map-reduce software Ø Managed by Apache foundation Ø Based on Google s Ø Google File System Ø MapReduce Ø Companies formed around Hadoop: Ø Cloudera Ø Hortonworks Ø MapR

68 Hadoop Ø Very active open source ecosystem Ø Several key technologies Ø HDFS: Hadoop File System Ø MapReduce: map-reduce compute framework Ø YARN: Yet another resource negotiator Ø Hive: SQL queries over MapReduce Ø

69 In-Memory Dataflow System Developed at the UC Berkeley AMP Lab M. Zaharia, M. Chowdhury, M. J. Franklin, S. Shenker, and I. Stoica. Spark: cluster computing with working sets. HotCloud 10 M. Zaharia, M. Chowdhury, T. Das, A. Dave, J. Ma, M. McCauley, M.J. Franklin, S. Shenker, I. Stoica. Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing, NSDI 2012

70 What Is Ø Parallel execution engine for big data processing Ø General: efficient support for multiple workloads Ø Easy to use: 2-5x less code than Hadoop MR Ø High level API s in Python, Java, and Scala Ø Fast: up to 100x faster than Hadoop MR Ø Can exploit in-memory when available Ø Low overhead scheduling, optimized engine

71 Spark Programming Abstraction Ø Write programs in terms of transformations on distributed datasets Ø Resilient Distributed Datasets (RDDs) Ø Distributed collections of objects that can stored in memory or on disk Ø Built via parallel transformations (map, filter, ) Ø Automatically rebuilt on failure Slide provided by M. Zaharia

72 RDD: Resilient Distributed Datasets Ø Collections of objects partitioned & distributed across a cluster Ø Stored in RAM or on Disk Ø Resilient to failures Ø Operations Ø Transformations Ø Actions

73 Operations on RDDs Ø Transformations f(rdd) => RDD Lazy (not computed immediately) E.g., map, filter, groupby Ø Actions: Triggers computation E.g. count, collect, saveastextfile

74 Example: Log Mining Load error messages from a log into memory, then interactively search for various patterns

75 Example: Log Mining Load error messages from a log into memory, then interactively search for various patterns Driver

76 Example: Log Mining Load error messages from a log into memory, then interactively search for various patterns lines = spark.textfile( hdfs://file.txt ) Driver

77 Example: Log Mining Load error messages from a log into memory, then interactively search for various patterns Base RDD lines = spark.textfile( hdfs://file.txt ) Driver

78 Example: Log Mining Load error messages from a log into memory, then interactively search for various patterns lines = spark.textfile( hdfs://file.txt ) errors = lines.filter(lambda s: s.startswith( ERROR )) Driver

79 Example: Log Mining Load error messages from a log into memory, then interactively search for various patterns Transformed RDD lines = spark.textfile( hdfs://file.txt ) errors = lines.filter(lambda s: s.startswith( ERROR )) Driver

80 Example: Log Mining Load error messages from a log into memory, then interactively search for various patterns lines = spark.textfile( hdfs://file.txt ) errors = lines.filter(lambda s: s.startswith( ERROR )) messages = errors.map(lambda s: s.split( \t )[2]) messages.cache() Driver messages.filter(lambda s: mysql in s).count()

81 Example: Log Mining Load error messages from a log into memory, then interactively search for various patterns lines = spark.textfile( hdfs://file.txt ) errors = lines.filter(lambda s: s.startswith( ERROR )) messages = errors.map(lambda s: s.split( \t )[2]) messages.cache() messages.filter(lambda s: mysql in s).count() Driver Action

82 Example: Log Mining Load error messages from a log into memory, then interactively search for various patterns lines = spark.textfile( hdfs://file.txt ) errors = lines.filter(lambda s: s.startswith( ERROR )) messages = errors.map(lambda s: s.split( \t )[2]) messages.cache() Driver Partition 1 messages.filter(lambda s: mysql in s).count() Partition 2 Partition 3

83 Example: Log Mining Load error messages from a log into memory, then interactively search for various patterns lines = spark.textfile( hdfs://file.txt ) errors = lines.filter(lambda s: s.startswith( ERROR )) messages = errors.map(lambda s: s.split( \t )[2]) messages.cache() messages.filter(lambda s: mysql in s).count() Driver tasks tasks Partition 1 tasks Partition 2 Partition 3

84 Example: Log Mining Load error messages from a log into memory, then interactively search for various patterns lines = spark.textfile( hdfs://file.txt ) errors = lines.filter(lambda s: s.startswith( ERROR )) messages = errors.map(lambda s: s.split( \t )[2]) messages.cache() messages.filter(lambda s: mysql in s).count() Driver Partition 1 Read HDFS Partitio Partition 3 Read HDFS Partition Partition 2 Read HDFS Partitio

85 Example: Log Mining Load error messages from a log into memory, then interactively search for various patterns lines = spark.textfile( hdfs://file.txt ) errors = lines.filter(lambda s: s.startswith( ERROR )) messages = errors.map(lambda s: s.split( \t )[2]) messages.cache() messages.filter(lambda s: mysql in s).count() Driver Cache 3 Partition 3 Process & Cache Data Cache 1 Partition 1 Process & Cache Data Cache 2 Partition 2 Process & Cache Data

86 Example: Log Mining Load error messages from a log into memory, then interactively search for various patterns lines = spark.textfile( hdfs://file.txt ) errors = lines.filter(lambda s: s.startswith( ERROR )) messages = errors.map(lambda s: s.split( \t )[2]) messages.cache() messages.filter(lambda s: mysql in s).count() Driver results Cache 3 Partition 3 results results Cache 1 Partition 1 Cache 2 Partition 2

87 Example: Log Mining Load error messages from a log into memory, then interactively search for various patterns lines = spark.textfile( hdfs://file.txt ) errors = lines.filter(lambda s: s.startswith( ERROR )) messages = errors.map(lambda s: s.split( \t )[2]) messages.cache() Driver Cache 1 Partition 1 messages.filter(lambda s: mysql in s).count() messages.filter(lambda s: php in s).count() Cache 3 Partition 3 Cache 2 Partition 2

88 Example: Log Mining Load error messages from a log into memory, then interactively search for various patterns lines = spark.textfile( hdfs://file.txt ) errors = lines.filter(lambda s: s.startswith( ERROR )) messages = errors.map(lambda s: s.split( \t )[2]) messages.cache() messages.filter(lambda s: mysql in s).count() messages.filter(lambda s: php in s).count() Driver tasks Cache 3 Partition 3 tasks tasks Cache 1 Partition 1 Cache 2 Partition 2

89 Example: Log Mining Load error messages from a log into memory, then interactively search for various patterns lines = spark.textfile( hdfs://file.txt ) errors = lines.filter(lambda s: s.startswith( ERROR )) messages = errors.map(lambda s: s.split( \t )[2]) messages.cache() messages.filter(lambda s: mysql in s).count() messages.filter(lambda s: php in s).count() Driver Cache 3 Partition 3 Process from Cache Cache 1 Partition 1 Process from Cache Cache 2 Partition 2 Process from Cache

90 Example: Log Mining Load error messages from a log into memory, then interactively search for various patterns lines = spark.textfile( hdfs://file.txt ) errors = lines.filter(lambda s: s.startswith( ERROR )) messages = errors.map(lambda s: s.split( \t )[2]) messages.cache() messages.filter(lambda s: mysql in s).count() messages.filter(lambda s: php in s).count() Driver results Cache 3 Partition 3 results results Cache 1 Partition 1 Cache 2 Partition 2

91 Example: Log Mining Load error messages from a log into memory, then interactively search for various patterns lines = spark.textfile( hdfs://file.txt ) errors = lines.filter(lambda s: s.startswith( ERROR )) messages = errors.map(lambda s: s.split( \t )[2]) messages.cache() Driver Cache 1 Partition 1 messages.filter(lambda s: mysql in s).count() messages.filter(lambda s: php in s).count() Cache your data è Faster Results Full-text search of Wikipedia 60GB on 20 EC2 machines 0.5 sec from mem vs. 20s for on-disk Cache 3 Partition 3 Cache 2 Partition 2

92 Abstraction: Dataflow Operators map filter groupby sort union join leftouterjoin rightouterjoin reduce count fold reducebykey groupbykey cogroup cross zip sample take first partitionby mapwith pipe save...

93 Abstraction: Dataflow Operators map filter groupby sort union join leftouterjoin rightouterjoin reduce count fold reducebykey groupbykey cogroup cross zip sample take first partitionby mapwith pipe save...

94 Language Support Python lines = sc.textfile(...) lines.filter(lambda s: ERROR in s).count() Scala val lines = sc.textfile(...) lines.filter(x => x.contains( ERROR )).count() Java JavaRDD<String> lines = sc.textfile(...); lines.filter(new Function<String, Boolean>() { Boolean call(string s) { return s.contains( error ); } }).count(); Standalone Programs Python, Scala, & Java Interactive Shells Python & Scala Performance Java & Scala are faster due to static typing