What is Data Warehouse like

Transcription

1 What is Data Warehouse like in the Big Data Era?

2 Sales (Asia) Data Warehouse Sales (US) ETL ETL Collects and organizes historical data from multiple sources Inventory Advertising ETL ETL So far Ø Star Schemas Ø Data cubes Ø OLAP Queries

3 Sales (Asia) Sales (US) Inventory Advertising Text/Log Data ETL ETL ETL ETL ETL? Photos & Videos Data Warehouse Collects and organizes historical data from multiple sources Ø How do we deal with semi-structured and unstructured data? Ø Do we really want to force a schema on load? It is Terrible!

4 Sales (Asia) How do we Sales clean and organize this (US) data? Inventory Advertising ETL ETL Depends on use Text/Log Data ETL ETL ETL? Photos & Videos Data Warehouse Collects and organizes historical data from multiple sources Ø How do we deal with semi-structured How do we load and process this data and unstructured data? in a relational system? Ø Do we really want to Depends force a on schema use on load? Can be difficult... Requires thought... It is Terrible!

5 Sales (Asia) Sales (US) Data Lake ET L Inventory ET L Advertising ET L Store a copy of all the data in one place in its original natural form Enable data consumers to choose how to transform and use data. ET L Text/Log Data * E? L T Photos & Videos Schema on Read Enabled by new Tools: Map-Reduce & Distributed Filesystems What could go wrong? It is Terrible! *Still being defined [Buzzword Disclaimer]

6 The Dark Side of Data Lakes Ø Cultural shift: Curate à Save Everything! Ø Noise begins to dominate signal Ø Limited data governance and planning Example: hdfs://important/joseph_big_file3.csv_with_json Ø What does it contain? Ø When and who created it? Ø No cleaning and verification à lots of dirty data Ø New tools are more complex and old tools no longer work Enter the data scientist

7 A Brighter Future for Data Lakes Enter the data scientist Ø Data scientists bring new skills Ø Distributed data processing and cleaning Ø Machine learning, computer vision, and statistical sampling Ø Technologies are improving Ø SQL over large files Ø Self describing file formats & catalog managers Ø Organizations are evolving Ø Tracking data usage and file permissions Ø New job title: data engineers

8 What you ve learnt? Ø Why Data Warehouse? Ø Why Data Lake? Ø Why Data Scientists? Ø Why Data Engineers?

9 How to improve query performance?

10 1 TB SELECT SUM(Sales) FROM Table WHERE Country = Canada 10 MB/s 1.2 Day

11 Idea 1: Pre-computation SELECT SUM(Sales) FROM Table WHERE Country = Canada 1.2 Million SELECT SUM(Sales) FROM Table WHERE Country = USA 2.4 Million SELECT SUM(Sales) FROM Table WHERE Country = China 1.8 Million 1. Which query should be precomputed? 2. What if data is updated? 3. Howto useprecomputed queryresults?

12 Idea 2: Parallel Database 1 GB 1 GB 1 GB... 1 GB 1000 machines SELECT SUM(Sales) FROM Table WHERE Country = Canada 1.7 mins

13 David DeWitt David DeWitt

14 Parallel DBMSs How to evaluate a parallel DBMS? How to architect a parallel DBMS? How to partition data in a parallel DBMS?

15 Parallel DBMSs How to evaluate a parallel DBMS? How to architect a parallel DBMS? How to partition data in a parallel DBMS?

16 Performance Metrics for Parallel DBMSs Speedup ü More processors à Higher speed Scaleup ümore processors à Can process more data

17 Linear v.s. Non-linear Speedup

18 Linear v.s. Non-linear Scaleup

19 Parallel DBMSs How to evaluate a parallel DBMS? How to architect a parallel DBMS? How to partition data in a parallel DBMS?

20 Three Architectures Shared Memory Shared Nothing Shared Disk

21 Shared Memory GPU

22 Shared Nothing Parallel DBMSs, MapReduce, Spark

23 Shared Disk Azure Data Warehouse

24 Three Architectures Shared Memory Shared Nothing Computation vs. Communication Trade-offs Shared Disk Economic Consideration

25 Parallel DBMSs How to evaluate a parallel DBMS? How to architect a parallel DBMS? How to partition data in a parallel DBMS?

26 Horizontal Data Partitioning Round Robin üj Load Balancing ül Bad Query Performance Range Partitioning üj Good for range/point queries ül Data Skew (i.e., Bad Load balancing) Hash Partitioning üj Good for point queries ül Hard to answer range queries

27 Summary How to improve query performance? ü Precomputation ü Parallelism Parallel DBMSs ü Evaluation metrics: Speedup and Scaleup ü Architecture: Shared-memory, shared-nothing, shared-disk ü DataPartition: Round Robin, RangePartitioning, Hash Partitioning

28 Sources Dan Suciu s CSE 444 slides, Spring UC Berkeley DS100 Fall 2017