Just add Magic. Enterprise Parquet. Jean-Pierre Dijcks Product Management, Big

Transcription

1

2 Just add Magic Enterprise Parquet Jean-Pierre Dijcks Product Management, Big

3 Program Agenda Context Enterprise Parquet Q&A 3

4 Context 4

5 Use Cases and Non-Use Cases The entre presentaton focuses on AnalyTcs Write once, read many Tmes The use cases involve file based data stores, like: Hadoop Distributed File System (HDFS) Object Stores 5

6 Metadata Metadata Metadata Metadata Sharing Metadata Sharing Metadata Metadata Metadata 6

7 Files Metadata Challenges Cause issue with: Security Fine-grained access controls in Object Stores and HDFS Data Governance / LegislaTve Pressure (GDPR etc.) Performance Performance penaltes in Schema on Read cases Agility ETL challenges on many changing file formats Cost Avoid file sprawl and storage explosion through copies to solve 7

8 Hive: Adding Metadata to enable SQL Hive Metastore Defines: SELECT name FROM my_cust WHERE id = 1 InputFormat RecordReader SerDe SQL ExecuTon /n /n /n /n Any File Type Create Splits Create Records Create Agributes Select Data 8

9 How does Apache Parquet Work? Create and Query Parquet Files Schema on Write Parquet implements a database storage structure with metadata and parsed data elements Columns Column ProjecTon Select name from my_cust where id = 1 Rows Predicate based Row EliminaTon Metadata for blocks Metadata drives database-like scanning behavior 9

10 How does Parquet Work? Create and Query Parquet Files Schema on Write Parquet implements a database storage structure with metadata and parsed data elements Schema on READ Rows Columns Column ProjecTon Select name from my_cust where id = 1 Schema on WRITE Predicate based Row EliminaTon Metadata for blocks Metadata drives database-like scanning behavior 10

11 Apache Parquet, solving one Performance Angle Columnar Database File Format Schema on Write: So all your data is parsed You must do ETL Some data elements will not fit Does track metadata and schema Columnar IO avoidance Beger performance Need to keep the original data for archive purposes 11

12 Enterprise Parquet 12

13 Enterprise Parquet Fine grained, in-file access control Columnar IO + Original Data Faster Schema-on-Read Full 1234 Masked 12XX Tokenized wxyz SQL Columnar IO,,,,,, UnZIP Full Original Data Schema on Write Hive SerDe Schema on Read Oracle SerDe Compress 13

14 Standard Use Case - Baseline Two Files = Apache Parquet + Text file Text.gz, Source Text in GZIP 2x Files on Disk ~1.6-2x Footprint Parquet Columnar Slow text-mode Ad-hoc queries Columnar Queries 14

15 Enterprise Parquet Enhancements Overview Single File, New Access Modes, Less Storage Space Text.gz, Source Text in GZIP Two Files on Disk ~1.6x - 2x space Parquet Columnar Single File on Disk ~1.05x space Parquet Columnar Parquet partton Backward CompaTble 15

16 Enterprise Parquet Enhancements Overview Single File, New Access Modes, Less Storage Space Text.gz, Source Text in GZIP Two Files on Disk ~1.6x - 2x space Parquet Columnar Parquet Columnar Parquet partton Backward CompaTble Hidden Enhanced Oracle Extras Enterprise Extra Single File on Disk ~1.05x space 16

17 Enterprise Parquet Enhancements Overview Single File, New Access Modes, Less Storage Space Text.gz, Source Text in GZIP Two Files on Disk ~1.6x - 2x space Single File on Disk ~1.05x space Parquet Columnar Parquet Columnar Slow text-mode Ad-hoc queries Columnar Queries CompaCble Mode: Parquet Queries 17

18 Enterprise Parquet Enhancements Overview Single File, New Access Modes, Less Storage Space Text.gz, Source Text in GZIP Two Files on Disk ~1.6x 2x space Single File on Disk ~1.05x space Parquet Columnar Parquet Columnar Enterprise Extra Slow text-mode Ad-hoc queries Columnar Queries CompaCble Mode: Parquet Queries Dynamic / Text Mode True Unzip Faster Ad-hoc Text queries 18

19 Enterprise Parquet Enhancements #1b RedacTon and Masking : New StaTc Mode Access Text.gz, Source Text in GZIP Parquet Columnar Key Enterprise Parquet Columnar Enterprise Extra StaCc Mode Enhanced Binary Access Redacted / Unredacted Values... 19

20 Ingest - Security Upon Ingest specific fields are marked to redact with details like: EncrypTon Key RedacTon Pagern RedacTon or tokenizaton columns Etc. 20

21 Ingest - Security 21

22 Ingest - AutomaTon Industry Standard and Documented files are ingested with a DefiniTon based on this Standard Example: HL7 Web Logs Trading Data Etc. 22

23 Ingest AutomaTon 23

24 Query Security Enterprise Parquet Redacted Privileged 24

25 Query Security 25

26 Unzip Parquet File 26

27 Unzip 27

28 High Performance Schema-on-Read Ingest as one schema, read as another schema Single file acts as many files 28

29 High Performance Schema-on-Read 5x Speedup 29

30 Summary Security Access Controls inside Files Performance Speed up like Parquet Storage No DuplicaTon of Data CompaTbility 100% Parquet CompaTble 30

31 To Do Finalize File Work for GA IntegraTon Big Data SQL and other SQL engines Key Management and AuthenTcaTon Frameworks Kaxa Pipelines Big Data Manager Autonomous Ingest for any format More GDPR s The right to be forgogen Autonomous RedacTon on SensiTve elements Data Provenance in Files Your Requirements here 31

32 QuesTons and Answers 32

33 33

34