Andreas Weininger,

Transcription

1 External Tables: New Options not just for Loading Data Andreas Weininger,

2 Agenda Why external tables? What are the alternatives? How to use external tables Caveats Netezza interface 2

3 Why External Tables? Fast loading and unloading Goal as fast or faster than LOAD/HPU Reduced logging Flexibility (usage like internal tables) New use cases Compatibility to other databases (Netezza) 3

4 How to create an external table CREATE EXTERNAL TABLE customer_ext LIKE customer USING ( DATAOBJECT '/mnt/blumeta0/home/bluadmin/customer.unl ); Meta data created for external table No files created 4

5 Create an External Table based on a SQL Statement CREATE EXTERNAL TABLE customer_ext '/mnt/blumeta0/home/db2inst1/customer.unl USING ( LOGDIR '/tmp ) AS SELECT * FROM customer 5

6 Anonymous External Table based on a SQL Statement CREATE EXTERNAL TABLE '/mnt/blumeta0/home/db2inst1/customer1.unl USING ( LOGDIR '/tmp' ) AS SELECT * FROM customer DB20000I The SQL command completed successfully. Unload files 6

7 Loading Data with External Tables INSERT INTO customer SELECT * FROM customer_ext DB20000I The SQL command completed successfully. 7

8 Log Files (1/3) ============================================================================= Load started at: 25-Sep-20 08:41:55 UTC Database: BLUDB Schema: BLUADMIN DefinerID: DB2INST1 Tablename: CUSTOMER_EXT Datafile: /mnt/blumeta0/home/db2inst1/customer.unl Host: ibmvm 8

9 Log Files (2/3) Load Options Field delimiter: ' ' NULL value: NULL File Buffer Size (MB): 8 Load Replay Region (MB): 0 Encoding: INTERNAL Max errors: 1 Skip records 0 Max rows: 0 FillRecord: No Truncate String: No Escape Char: None Accept Control Chars: No Allow CR in string: No Ignore Zero: No Quoted data: NO Require Quotes: No BoolStyle: 1_0 Decimal Delimiter: '.' Disable NFC: No Date Style: YMD Date Delim: '-' Y2Base: 2000 Time Style: 24HOUR Time Delim: ':' Time extra zeros: No Date Format: (null) Time Format: (null) TimeStamp Format: (null) CCSID : Record Delim: 9

10 Log Files (3/3) Statistics number of records read: 3000 number of bytes read: number of records skipped 0 number of bad records number of records loaded: 3000 Elapsed Time (sec): Load completed at: 25-Sep-20 08:41:55 UTC 10

11 Unloading Data with External Tables INSERT INTO customer_ext SELECT * FROM customer DB20000I The SQL command completed successfully. 11

12 Object Store Support (Amazon S3) CREATE EXTERNAL TABLE exttab2(a int) using (dataobject 'datafile2.dat' s3('s3.amazonaws.com', 'XXXOS :xxx123456, URL of S3 compatible web service S3 access key id 'bs07c6e d92174e5e96d5be9382aa4c33xxx5a14eaa9172c70f8 df16', 'my_dev' ) ) S3 bucket S3 secret key 12

13 Object Store Support (IBM Cloud S3) CREATE EXTERNAL TABLE exttab2(a int) using ( dataobject 'datafile2.dat s3('s3-api.us-geo.objectstorage.softlayer.net, '1a2bkXXXsaddntLo0xX0, S3 access key id 'XXxxiEPjJ7T7WBUz74E6abcdABCDE8Q7RgU4gYY9, URL of S3 compatible web service ) 'my_dev ) S3 bucket S3 secret key 13

14 Object Store Support (SWIFT) CREATE EXTERNAL TABLE exttab1(a int) using ( dataobject 'datafile1.dat swift(' ) 'XXXOS :xxx123456', 'b207c6e d92174esdf6d5be9382aa4c335945a14eaa9172c70f8df16', 'my_dev' ) bucket password access id / user URL 14

15 Selecting from empty external table SELECT count(*) FROM customer_ext SQL20569N The external table operation failed due to a problem with the corresponding data file. File name: "/mnt/blumeta0/home/db2inst1/customer.unl". Reason code: "1". SQLSTATE=428IB $ touch /mnt/blumeta0/home/db2inst1/customer.unl SELECT count(*) FROM customer_ext record(s) selected. 15

16 Error Handling (1/2) Found bad records bad #: input row #(byte offset to last char examined) [field #, declaration] diagnostic, "text consumed" [last char examined] : 1(33)[8, VARCHAR(10)] expected end of row, "0 "[1] 2: 2(40)[8, VARCHAR(10)] expected end of row, "0 "[6] 3: 3(41)[8, VARCHAR(10)] expected end of row, "0 "[2] 16

17 Error Handling (2/2) Statistics number of records read: 3 number of bytes read: number of records skipped 0 number of bad records number of records loaded: 0 Elapsed Time (sec): 1.0 Rows/Second : 3.0 Bytes/Second : Load completed at: 25-Sep-20 13:38:55 UTC 17

18 Use Case: Analytics on Flat Files SELECT c_nation, count(*) cnt FROM customer_ext WHERE c_region = 'AMERICA GROUP BY c_nation C_NATION CNT ARGENTINA 120. BRAZIL 135. CANADA 125. PERU 127. UNITED STATES record(s) selected. 18

19 Use Case: Analytics on Flat Files (Cont): Joins SELECT c_nation, count(*) cnt FROM customer_ext, lineorder_ext WHERE c_custkey = lo_custkey AND c_region = 'AMERICA GROUP BY c_nation C_NATION CNT ARGENTINA BRAZIL CANADA PERU UNITED STATES record(s) selected. 19

20 How to analyze performance (db2exfmt) (1/7) Original Statement: select c_nation, count(*) cnt from customer_ext, lineorder_ext where c_custkey = lo_custkey and c_region = 'AMERICA' group by c_nation Original Statement: select c_nation, count(*) cnt from customer, lineorder where c_custkey = lo_custkey and c_region = 'AMERICA' group by c_nation 20

21 How to analyze performance (db2exfmt) (2/7) Optimized Statement: SELECT Q4.C_NATION AS "C_NATION", Q4.$C1 AS "CNT" FROM (SELECT Q3.C_NATION, COUNT_BIG(*) FROM (SELECT Q2.C_NATION FROM BLUADMIN.LINEORDER_EXT AS Q1, BLUADMIN.CUSTOMER_EXT AS Q2 WHERE (Q2.C_REGION = 'AMERICA') AND (Q2.C_CUSTKEY = Q1.LO_CUSTKEY) ) AS Q3 GROUP BY Q3.C_NATION ) AS Q4 Optimized Statement: SELECT Q4.C_NATION AS "C_NATION", Q4.$C1 AS "CNT" FROM (SELECT Q3.C_NATION, COUNT_BIG(*) FROM (SELECT Q2.C_NATION FROM BLUADMIN.LINEORDER AS Q1, BLUADMIN.CUSTOMER AS Q2 WHERE (Q2.C_REGION = 'AMERICA') AND (Q2.C_CUSTKEY = Q1.LO_CUSTKEY) ) AS Q3 GROUP BY Q3.C_NATION ) AS Q4 21

22 How to analyze performance (db2exfmt) (3/7) Access Plan: Total Cost: Query Degree: 4 Rows RETURN ( 1) Cost I/O GRPBY ( 2) LMTQ ( 3) TBSCAN ( 4) SORT ( 5) e+06 pgrpby ( 6) e+07 HSJOIN ( 7) / \ TBSCAN TBSCAN ( 8) ( 9) TABLE: BLUADMIN TABLE: BLUADMIN LINEORDER_EXT CUSTOMER_EXT Q1 Q2 Access Plan: Total Cost: Query Degree: 4 Rows RETURN ( 1) Cost I/O 25 LTQ ( 2) CTQ ( 3) GRPBY ( 4) ^HSJOIN ( 5) / \ TBSCAN TBSCAN ( 6) ( 7) CO-TABLE: BLUADMIN CO-TABLE: BLUADMIN LINEORDER CUSTOMER Q1 Q2 22

23 How to analyze performance (db2exfmt) (4/7) Extended Diagnostic Information: Diagnostic Identifier: 1 Diagnostic Details: EXP0020W Table has no statistics. The table "BLUADMIN"."LINEORDER_EXT" has not had runstats run on it. This may result in a sub-optimal access plan and poor performance. Diagnostic Identifier: 2 Diagnostic Details: EXP0020W Table has no statistics. The table "BLUADMIN"."CUSTOMER_EXT" has not had runstats run on it. This may result in a sub-optimal access plan and poor performance. 23

24 How to analyze performance (db2exfmt) (5/7) Objects Used in Access Plan: Schema: BLUADMIN Name: CUSTOMER_EXT Type: Table Time of creation: Last statistics update: Number of columns: 8 Number of rows: Width of rows: 58 Number of buffer pool pages: 3 Number of data partitions: 1 Distinct row values: No Source for statistics: Single Node Prefetch page count: 0 Container extent page count: 0 Table overflow record count: 0 Table Active Blocks: -1 Average Row Compression Ratio: -1 Percentage Rows Compressed: -1 Average Compressed Row Size: -1 24

25 How to analyze performance (db2exfmt) (6/7) Schema: BLUADMIN Name: LINEORDER_EXT Type: Table Time of creation: Last statistics update: Number of columns: 17 Number of rows: Width of rows: 34 Number of buffer pool pages: 3 Number of data partitions: 1 Distinct row values: No Source for statistics: Single Node Prefetch page count: 0 Container extent page count: 0 Table overflow record count: 0 Table Active Blocks: -1 Average Row Compression Ratio: -1 Percentage Rows Compressed: -1 Average Compressed Row Size: -1 25

26 How to analyze performance (db2exfmt) (7/7) Name: Type: Schema: BLUADMIN LINEORDER Column-organized Table Time of creation: Last statistics update: Number of columns: 17 Number of rows: Width of rows: 34 Number of buffer pool pages: 1448 Number of data partitions: 1 Distinct row values: No Tablespace name: USERSPACE1 Tablespace overhead: Tablespace transfer rate: Source for statistics: Single Node Prefetch page count: 4 Container extent page count: 4 Table overflow record count: 0 Table Active Blocks: -1 Average Row Compression Ratio: -1 Percentage Rows Compressed: -1 Average Compressed Row Size: -1 26

27 Runstats on External Table? $ db2 runstats on table lineorder_ext SQL5115N The command or statement was not executed because the following functionality is not supported in the current environment: "TABLE RUNSTATS". SQLSTATE=

28 Use Case: Pseudonymization and Anonymization (1/2) CREATE EXTERNAL TABLE customer_p_ext ); LIKE customer USING ( DATAOBJECT '/mnt/blumeta0/home/db2inst1/customerp.unl 28

29 Use Case: Pseudonymization and Anonymization (2/2) INSERT INTO customer_p_ext SELECT c_custkey, p_pname, c_address, c_city, c_nation, c_region, 'xx-xxx-xxx-' substr(c_phone,12,4), c_mktsegment FROM customer, pseudonym WHERE c_name = p_name 29

30 Use Case: ELT (Constraint Check) (1/3) Problem description: New data should be loaded into fact table lineorder, but only if customer already exists in dimension table customer, otherwise row should be put in exception table for later processing CREATE EXTERNAL TABLE lineorder_delta_ext ( LO_LINENUMBER BIGINT, LO_NAME VARCHAR(25), LO_ORDERDATE INTEGER, LO_ORDERPRIOTITY VARCHAR(15), LO_SHIPPRIOTITY INTEGER, Used for check whether customer exists ) USING ( DATAOBJECT '/mnt/blumeta0/home/db2inst1/lineorder_delta.unl' ) 30

31 Use Case: ELT (Constraint Check) (2/3) Put non matching rows in exception table: INSERT INTO lineorder_no_matching_cust SELECT * FROM lineorder_delta_ext WHERE NOT EXISTS ( SELECT 1 FROM CUSTOMER WHERE LO_NAME = C_NAME ) 31

32 Use Case: ELT (Constraint Check) (3/3) Put rows which have already matching row in the dimension table into target table: INSERT INTO lineorder_fact SELECT LO_LINENUMBER, C_CUSTKEY, LO_ORDERDATE, LO_ORDERPRIOTITY, LO_SHIPPRIOTITY, FROM lineorder_delta_ext, customer WHERE LO_NAME = C_NAME Join of external with internal table 32

33 Using compressed files CREATE EXTERNAL TABLE customer_comp_ext LIKE customer USING ( DATAOBJECT '/mnt/blumeta0/home/db2inst1/customer-comp.unl.gz COMPRESS GZIP ); INSERT INTO customer_comp_ext SELECT * FROM customer; SELECT count(*) FROM customer_comp_ext $ ls -la customer-comp.unl.gz customer.unl -rw-r db2inst1 db2iadm Sep 26 06:43 customer-comp.unl.gz -rw-r--r-- 1 db2inst1 db2iadm Sep 25 08:54 customer.unl 33

34 Using named pipes $ mkfifo /mnt/blumeta0/home/db2inst1/customer.pipe $ cat < /mnt/blumeta0/home/db2inst1/customer.pipe CREATE EXTERNAL TABLE customer_pipe_ext LIKE customer USING ( DATAOBJECT '/mnt/blumeta0/home/db2inst1/customer.pipe ) DB20000I The SQL command completed successfully. INSERT INTO customer_pipe_ext SELECT * FROM customer DB20000I The SQL command completed successfully. 34

35 Caveats Load files must in path below path specified in extbl_location Otherwise files cannot be loaded 35

36 DML Compatibility: External Tables Support Netezza syntax and options Extended with options more appropriate to Db2. New options like CCSID should be used for IIAS<>IIAS Support both LOCAL and REMOTE In cloud also supports Object Store (considered a LOCAL path) File formats GZIP files can be loaded GZIP files can be created on UNLOAD using COMPRESS(GZIP) REMOTE: The network traffic can be compressed using GZIP or LZ4 Do not use if file is already GZIP compressed 36

37 DML Compatibility: External Tables Best Practices Unloading Generally specify CCSID 1208 to keep data in Unicode. Use ENCODING(INTERNAL) only if unloading to move to Netezza LOCAL Consider PARTITION ALL to increase performance by unloading each MLN (dataslice) into its own file. REMOTE Use LZ4 compression for most efficient network exchange Alternately use COMPRESS GZIP to produce a GZIP file on remote source 37

38 DML Compatibility: External Tables Best Practices Loading Use default ENCODING only if files originate from Netezza Specify CCSID if loading from other sources including IIAS LOCAL If multiple files consider using PARTITION [ALL 1-N] to load files in parallel using single statement REMOTE Use LZ4 compression for most efficient network exchange 38

39 DML Compatibility: External Tables Tools Anything that can issue SQL CLPPlus / Db2 CLP / jsqsh / for Loading Unloading Other use cases Netezza compatible syntax dbload command line tool db_migrate Data migration from Netezza to IIAS 39

40 DML Compatibility: External Table REMOTESOURCE Named External Table 40 create external table external_foo (c1 int, c2 int) using ( Load dataobject ( /tmp/foo.txt ) REMOUTESOURCE LZ4); insert into foo (select * from external_foo); Unload insert into external_foo (select * from foo); Transient Load insert into foo select * from external /tmp/foo.txt using (REMOUTESOURCE LZ4 delimiter, ); Transient Unload create external table /tmp/foo.txt using (REMOUTESOURCE LZ4 delimiter, ) as select * from foo;

41 Questions? 41

42 Thank you ITALIAN HINDI FRENCH JAPANESE BRAZILIAN PORTUGUESE SIMPLIFIED CHINESE TRADITIONAL CHINESE SPANISH RUSSIAN TAMIL THAI GERMAN ARABIC 42