Big Data SQL Deep Dive

Transcription

1

2 Big Data SQL Deep Dive Jean-Pierre Dijcks Big Data Product Management DOAG 2016 Copyright 2016, Oracle and/or its affiliates. All rights reserved. 2

3 Safe Harbor Statement The following is intended to outline our general product direclon. It is intended for informalon purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or funclonality, and should not be relied upon in making purchasing decisions. The development, release, and Lming of any features or funclonality described for Oracle s products remains at the sole discrelon of Oracle. 3

4 The Best of Both Worlds SQL Big Data SQL 4

5 Big Data SQL s Unique Architecture 1 Consolidate Metadata across Silos 2 Enables seamless Blending of any Type of Data 3 Unique Architecture to deliver Performance: OpLmize end-to-end execulon of analylcal queries Pushing down Smart processing into diverse data Lers 5

6 Session Content Using Metadata to Read ANY Data Query Flow IO OpLmizaLons Join OpLmizaLons CerLficaLon Update Q&A 6

7 Big Data SQL Using Metadata to Read Data Copyright 2016, Oracle and/or its affiliates. All rights reserved. 7

8 How Data is Stored in HDFS data hr salaries.csv website clicks1.json clicks2.json clicks3.json clicks4.json Hanks,Spielberg, Spielberg,Cameron, Cameron,Oprah, Oprah,Boss, {"custid": ,"movieid":1948,"genreid":9,"lme":" :00:00:22 } {"custid": ,"movieid":null,"genreid":null,"lme":" :00:00:26 } {"custid": ,"movieid":11547,"genreid":44,"lme":" :00:00:32 } {"custid": ,"movieid":11547,"genreid":44,"lme":" :00:00:42 } 8

9 Organize and Describe Data with Hive hive data hr warehouse salaries.csv Database data.db website Table hr salaries.csv clicks1.json Table website clicks2.json month= ParLLon clicks1.json clicks3.json clicks2.json clicks4.json month= ParLLon clicks3.json clicks4.json InformaLon is captured in Hive Metastore HDFS Folders become: Databases Tables ParLLons Table includes metadata for parsing files using Java classes InputFormat defines chunks called splits based on file type RecordReader creates rows out of splits SerDe creates columns 9

10 How does Hive read ANY data? Hive Metastore Defines: SELECT name FROM my_cust WHERE id = 1 InputFormat RecordReader SerDe SQL ExecuLon /n /n /n /n Any File Type Create Splits Create Records Create Aqributes Select Data 10

11 Metadata: Extend Oracle External Tables CREATE TABLE movielog ( click VARCHAR2(4000)) ORGANIZATION EXTERNAL ( TYPE ORACLE_HIVE DEFAULT DIRECTORY DEFAULT_DIR ACCESS PARAMETERS ( com.oracle.bigdata.tablename logs com.oracle.bigdata.cluster mycluster )) REJECT LIMIT UNLIMITED; New types of external tables ORACLE_HIVE (leverage hive metadata) ORACLE_HDFS (specify metadata) Access parameters used to describe how to idenlfy sources and process data on the hadoop cluster 11

12 Access Parameters: HDFS Example CREATE TABLE WEB_SALES_CSV ( WS_SOLD_DATE_SK NUMBER, WS_SOLD_TIME_SK NUMBER, WS_ITEM_SK NUMBER ) ORGANIZATION EXTERNAL ( TYPE ORACLE_HDFS DEFAULT DIRECTORY DEFAULT_DIR ACCESS PARAMETERS ( com.oracle.bigdata.cluster=orabig com.oracle.bigdata.fileformat=textfile com.oracle.bigdata.rowformat: DELIMITED FIELDS TERMINATED BY ' com.oracle.bigdata.erroropt: {"aclon": "replace", "value": "-1"} ) LOCATION ('/data/tpcds/benchmarks/bigbench/data/web_sales') ) REJECT LIMIT UNLIMITED; Access Parameters describe source data and processing rules Schema-on-Read 12

13 Access Parameters: ORACLE_HIVE CREATE TABLE WEB_SALES_CSV ( WS_SOLD_DATE_SK NUMBER, WS_SOLD_TIME_SK NUMBER, WS_ITEM_SK NUMBER ) ORGANIZATION EXTERNAL ( TYPE ORACLE_HIVE DEFAULT DIRECTORY DEFAULT_DIR ACCESS PARAMETERS ( com.oracle.bigdata.cluster=orabig com.oracle.bigdata.tablename: csv.web_sales com.oracle.bigdata.erroropt: {"aclon": "replace", "value": "-1"} com.oracle.bigdata.datamode=automalc ) REJECT LIMIT UNLIMITED; Access Parameters refer to metadata in Hive Add processing rules 13

14 Viewing Hive Metadata from Oracle Database ALL_HIVE_DATABASES, ALL_HIVE_TABLES, ALL_HIVE_COLUMNS ALL_HIVE_COLUMNS 14

15 CreaLng Tables SQL Developer with Hive JDBC Right-click on Hive Table. Use in Oracle Big Data SQL Review generated columns. Update as needed - focusing on data types and precision Add oplonal access parameters. AutomaLcally generate table or save DDL. See: hqps://blogs.oracle.com/datawarehousing/entry/oracle_sql_developer_data_modeler 15

16 Big Data SQL on top of Hive Data Hive Metastore InputFormat RecordReader SerDe Big Data SQL /n /n /n /n Any File Type Create Splits Create Records Create Aqributes Convert Data & Smart Scan 16

17 Recommended Approach Use ORACLE_HIVE When Possible Oracle Database query execulon accesses Hive metadata at describe Lme Changes to underlying Hive access parameters will not impact Oracle table one exceplon column list and parllon lists in 12.2 Metadata an enabler for performance oplmizalons ParLLon pruning and predicate pushdown into intelligent sources ULlize tooling for simplified table definilons SQL Developer and DBMS_HADOOP packages 17

18 Big Data SQL Query Flow Copyright 2016, Oracle and/or its affiliates. All rights reserved. 18

19 SQL-on-Hadoop Engines Share Metadata, not MapReduce Hive Metastore Oracle Big Data SQL SparkSQL Hive Impala Hive Metastore Table DefiniLons: movieapp_log_json Tweets avro_log Metastore maps DDL to Java access classes Copyright 2015, Oracle and/or its affiliates. All rights reserved. 19

20 A Big Data SQL Query Apply SIs Query Get ParLLons Get Block List Smart no Fetch Data yes Push Predicates Distribute Processing Convert Data Types Project Columns Rows no Create SIs Return Results Filter (Smart Scan) yes Describe Fetch 20

21 Big Data SQL IO OpTmizaTons Copyright 2016, Oracle and/or its affiliates. All rights reserved. 21

22 Big Data SQL Performance Features IO ReducTon Features Deliver Compound Results User Query 100 TB ParLLon Pruning 10 TB Storage Indexing 1 TB Predicate Pushdown 100 GB 22

23 How does Hive read data FASTER? Hive Metastore SELECT name FROM my_cust WHERE id = 1 Use parllon pruning to reduce IO /n /n /n /n Define more fine-grained schema on files: ParLLoning Create Splits Create Records Create Aqributes Select Data 23

24 ParLLon Pruning and a Big Data SQL Query Query Get ParLLons Get Block List ParLLon pruning reduces the number of parllons that are going to be scanned Because only a subset of parllons are scanned the block list shrinks equally Distribute Processing Project Columns IO ReducLon Describe Fetch 24

25 Recommended Approach Use Hive ParTTons when Possible As with Oracle Database, using parllons can dramalcally drive down response Lmes for Big Data SQL queries (as it does for Hive) Schema design and understanding analylcs paqerns are slll beneficial But don t go overboard 25

26 Big Data SQL Storage Index Example: Find revenue for movies in a category 9 (Comedy) HDFS Min 1 Max 3 Min 12 Max 35 Big Data SQL Storage Index works on HDFS Chunks and creates an SI for each chunk Min / max value is recorded for columns included in a storage index (max # of colums = 32) Movies.json Min 7 Max 10 Storage index provides parllon pruning like performance for unmodeled data sets 26

27 Storage Indexing and a Big Data SQL Query Query Get ParLLons Get Block List Distribute Processing Project Columns Apply SIs Storage Indexes, before scanning data, further trims down the block list, reducing IO more based on the parllons IO ReducLon Describe Fetch 27

28 Storage Indexes Things to Think About Performance Cost vs. Benefits An SI is created when a block has not returned any data from predicates This is a second read of the data in the query causing inilal query performance to see performance impact StaLsLcal Relevance: HDFS based data is not parsed, and it is collected in large blocks (BDA = 256MB) An SI is maintained on that block, making the stalslcal chance of filter column values not in the block lower RecommendaLon: Cluster Data if possible or ascertain presence of natural transaclon ordering 28

29 How does Parquet Work? Create and Query Parquet Files Schema on Write Parquet implements a database storage structure with metadata and parsed data elements Columns Column ProjecLon Select name from my_cust where id = 1 Rows Predicate based Row EliminaLon Metadata for blocks Metadata drives database-like scanning behavior 29

30 Parquet Benefits Much faster reads because parsing penalty is no longer incurred Schema and data metadata enables parsing, IO skipping etc. providing oplmizalons like done in Database files Columnar format, enables IO oplmizalons for AnalyLcs Draw-backs Schema on write reduces the flexibility of schema on read and makes this just like a database Increases size of data stored as data is now replicated into oplmized formats Columnar formats does penalize IO that is row focused 30

31 A Big Data SQL Query Smart no yes Push Predicates Query predicates are handed to the Parquet reader classes which use columnar formats and metadata to eliminate IO only reading the subset of data requested Fetch Data IO ReducLon Describe Fetch 31

32 Parquet (or ORC) Things to Think About Performance Cost vs. Benefits In order to get the performance benefits, data has to parsed (= Inserted) This is a separate step, coslng Lme and increasing complexity When to use Parquet (or ORC) Do: Data Mart like constructs that are long lived Do: StaLc data, not too much changes in your schemas Don t: ETL like jobs that transform data in transient pipelines Don t: Very frequent changes to structures All the rules around schema evolulon in Databases apply 32

33 Smart Scan Smart no Fetch Data Convert Data Types Smart Scan reduces IO that is sent to the database engine It works on any data format CSV and other dumb formats are most affected But if ORC or Parquet deliver extra data, Smart Scan will cut it down to precise size Filter (Smart Scan) 33

34 A Big Data SQL Query Query Get ParLLons Apply SIs Get Block List Prq/ orc no yes Push Predicates Return Results Distribute Processing Project Columns Fetch Data Convert Data Types Filter (Smart Scan) No Data Fetch Data Create SIs Describe Fetch 34

35 Big Data SQL Join OpTmizaTons Copyright 2016, Oracle and/or its affiliates. All rights reserved. 35

36 Join OpLmizaLon with Bloom Filters A Bloom filter is a low-memory data structure that tests membership in a set correctly indicates when an element is *not* in a set. There could be false posilves is used to filter data in BDS cells - especially when joining large facts to small dimension tables OpLmizer will automalcally ullize Bloom filters to improve performance Can be influenced by PX_JOIN_FILTER/NO_PX_JOIN_FILTER hints Bloom filters are created in Database, so data does flow from HDFS to Database, but impact tends to be very small 36

37 Join OpLmizaLon with Bloom Filters Oracle Database Store state = CA floorspace > Create Join Filter s.store = ss.store BDS Cell Store Sales list_price > 100 Use Join Filter Apply Other Filters Query joins store and store sales table Bloom filter (bit vector) is created based on the join column Bloom filter & other filters are applied on the BDS Cell Is this store in the bit vector? PotenLal massive reduclon in data returned to database for join 37

38 Join OpLmizaLon Example: SELECT st.s_manager, st.s_hours, s.ss_list_price, s.ss_sales_price, s.ss_ext_discount_amt FROM store_sales s, store_orcl st WHERE s.ss_store_sk = st.s_store_sk AND st.s_state='ca' AND st.s_floor_space > AND s.ss_list_price > 100; 38

39 Big Data SQL on HBase (and ORC for fun) Business queston: I need a report of all sales by Lme for each job role (posilon) SQL> SELECT e.position, d.d_year, SUM(s.ss_ext_wholesale_cost), FROM store_sales_orc s, emp_hbase e, date_dim d WHERE e.rowkey = s.ss_customer_sk AND s.ss_sold_date_sk =d. d_date_sk AND e.rowkey > 0 GROUP BY e.position, d.d_year

40 Big Data SQL on HBase (and ORC for fun)

41 Big Data SQL CerTficaTon Update Copyright 2016, Oracle and/or its affiliates. All rights reserved. 41

42 Today s On-Premises Deployment Models 42

43 Announcing More Deployment OpLons for Big Data SQL 43

44 44

45