Start Working with Parquet!!!!

Transcription

1

2 My Goal Tonight. Start Working with Parquet!!!!

3 Parquet Query Performance Origin of Parquet Parquet Storage Query Request Usage with Hadoop Tools Customer Examples

4 Topics Parquet Defined Storage & Encoding Query Performance Parquet & Hadoop Tools Customer Success Story

5 Parquet Defined File Format Column- Based Open Framework Used by Many Hadoop Tools Open Source (Apache Incubator) Based on Google Dremel paper

6 Early Development (parquet.io)

7 Twi[er 500 million tweets/day; massive data growth Compressed data for storage, but Need to Analyze all data Product Manager: always improving site features

8 Row- based storage From h[ps://blog.twi[er.com/2013/dremel- made- simple- with- parquet

9 Column- based storage From h[ps://blog.twi[er.com/2013/dremel- made- simple- with- parquet

10 Encoding Each Column: Homogeneous Data Delta: cmestamps, integers Prefix: Delta for Strings Dicconary: Small Sets (i.e. State Codes) Run Length Encoding: Repeccve Data Example: Delta, 16x 32x encoding

11 Key Concepts Row Group: Columns of x file rows stored together Column Chunk: Data for a Particular Column. Contiguous Pages: Subset of Column Values within a Column Chunk

12 Queries on Parquet Compared to Row- based Storage, Column- based Storage minimizes Seeks Seek a column, read the whole column Vs. Seek- skip or seek- read of individual data values.

13 Query Request: Select A, C from X Table Row-Based Storage: Seeks => Read rows, drop unselected columns Column-Based Storage: Seeks => # of columns

14 Stacsccs: Filter and Query Opcmizacon

15 Hadoop Tools and Frameworks Open Plalorm Several Hadoop Tools can use Parquet Assumes New Tools Language Agnoscc

16 Hadoop Tools and Frameworks

17 Current Hadoop Tools Query Tools: Hive, Impala, HAWQ, IBM Big SQL, Drill, Tajo, Pig, Presto Frameworks: Spark, MapReduce, Cascading, Crunch, Scalding, Kite Data Models: Avro, Thrin, ProtocolBuffers, POJOs

18 Hive hive> create table parquet_table_name (x INT, y STRING) ROW FORMAT SERDE 'parquet.hive.serde.parquethiveserde' STORED AS INPUTFORMAT "parquet.hive.deprecatedparquetinputformat" OUTPUTFORMAT "parquet.hive.deprecatedparquetoutputformat";

19 Pig grunt> store A into '/test-warehouse/tinytable' USING parquet.pig.parquetstorer

20 Impala [localhost:21000] > create table parquet_table (x int, y string) stored as parquet; [localhost:21000] > insert into parquet_table select x, y from some_other_table; Inserted rows in 33.52s [localhost:21000] > select y from parquet_table where x between 70 and 100;

21

22 Customer Success - Allstate Started as a Tableau/Impala PoC Tescng 800 columns, 1.1 Billion rows 2.3TB Raw CSV file Converted to Parquet: 106 GB!!! Row Count Query: Hive = ~ 8 min Impala = 171 s Impala & Parquet = 1.94s!!!

23 Parquet = Powerful Performance! Origin of Parquet Parquet Storage Query Request Usage with Hadoop Tools Customer Examples

24 Geung Started Parquet Documentacon h[p://parquet.incubator.apache.org/ Twi[er Engineering Blog Cloudera Developer Blog Google s Dremel Paper

25 Thank You! Chad