Data Compression in Oracle

Transcription

1 Data Compression in Oracle Carl Dudley University of Wolverhampton, UK UKOUG SIG Director 1

2 Main Topics Oracle 9i and 10g Compression - major features Compression in data warehousing Sampling the data to predict compression Pre-sorting the data for compression Behaviour of DML/DDL on compressed tables Compression Internals Advanced Compression in Oracle11g (for OLTP operations) Shrinking unused space 2

3 Oracle Data Compression Main Features 3

4 Compression : Characteristics Trades Physical I/O against CPU utilization Transparent to applications Can increase I/O throughput and buffer cache capacity Useful for 'read mostly' applications Decision Support and OLAP Compression is performed only when Oracle considers it worthwhile Depends on column length and amount of duplication Compression algorithms have caused little change to the Kernel code Modifications only to block formatting and accessing rows and columns No compression within individual column values or across blocks Blocks retrieved in compressed format in the buffer cache 4

5 Getting Compressed Building a new compressed table CREATE TABLE <table_name>... COMPRESS; CREATE TABLE <table_name> COMPRESS AS SELECT... Altering an existing table to be compressed ALTER TABLE <table_name> MOVE COMPRESS; No additional copy created but temp space and exclusive table level lock required for the compression activity ALTER TABLE <table_name> COMPRESS; Future bulk inserts may be compressed existing data is not Compressing individual partitions ALTER TABLE <table_name> MOVE PARTITION <partition_name> COMPRESS; Existing data and future bulk inserts compressed in a specific partition 5

6 Tablespace Level Compression Entire tablespaces can compress by default CREATE ALTER TABLESPACE < tablespace_name> DEFAULT [ COMPRESS NOCOMPRESS ]... All objects in the tablespace will be compressed by default 6

7 Compressing Table Data Uncompressed conventional emp table CREATE TABLE emp (empno NUMBER(4),ename VARCHAR2(12)...); Compressed emp table CREATE TABLE emp (empno NUMBER(4),ename VARCHAR2(12)...) COMPRESS; Could consider sorting the data on columns which lend themselves to compression 7

8 Table Data Compression Uncompressed emp table 7369 CLERK ACCOUNTING 7782 MANAGER PLANT 7902 ANALYST OPERATIONS 7900 CLERK OPERATIONS 7934 CLERK ACCOUNTING 7654 PLANT RESEARCH Compressed emp table Similar to index compression [SYMBOL TABLE] [A]=CLERK, [B]=ACCOUNTING, [C]=PLANT, [D]=OPERATIONS 7369 [A] [B] 7782 MANAGER [C] 7902 ANALYST [D] 7900 [A] [D] 7934 [A] [B] 7654 [C] RESEARCH 8

9 Observing Compression Information Compressed tables are not identified in user_tables in Oracle9i Fixed in Oracle10g Create a new view called my_user_tables by modifying the create view statement for user_tables in the catalog.sql file (see italic type) create or replace view MY_USER_TABLES (TABLE_NAME, TABLESPACE_NAME, CLUSTER_NAME, IOT_NAME, PCT_FREE, PCT_USED, : GLOBAL_STATS, USER_STATS, DURATION, SKIP_CORRUPT, MONITORING, CLUSTER_OWNER, DEPENDENCIES,COMPRESSED) as select o.name, decode(bitand(t.property, ), 0, ts.name, null), decode(bitand(t.property, 1024), 0, null, co.name), : decode(bitand(t.flags, ), , 'ENABLED', 'DISABLED'), decode(bitand(s.spare1, 2048), 2048, 'ENABLED', 'DISABLED') from sys.ts$ ts, sys.seg$ s, sys.obj$ co, sys.tab$ t, sys.obj$ o, sys.obj$ cx, sys.user$ cu : Create a public synonym for the new view and grant public access to it 9

10 Table Data Compression (continued) Scanning the compressed table is much faster Compressing can significantly reduce disk I/O - good for queries Possible increase in CPU activity Must unravel the compression but logical reads will be reduced SELECT table_name,compressed,num_rows FROM my_user_tables; TABLE_NAME COMPRESSED NUM_ROWS EC DISABLED EC1 ENABLED SELECT COUNT(ename) uncompressed FROM ec; UNCOMPRESSED Elapsed: 00:00:04.00 SELECT COUNT(ename) compressed FROM ec1; COMPRESSED Elapsed: 00:00:

11 Space Reduction Due to Compression Space usage summary Statistics for table EC Free Blocks...1 Total Blocks Total Bytes Unused Blocks...58 Unused Bytes Last Used Ext FileId...1 Last Used Ext BlockId Last Used Block...70 Statistics for table EC1 Free Blocks...0 Total Blocks Total Bytes Unused Blocks...46 Unused Bytes Last Used Ext FileId...1 Last Used Ext BlockId Last Used Block...82 Summary routine adapted from Tom Kyte s example 11

12 Compression in Data Warehousing 12

13 Compression in Data Warehousing Fact tables are good candidates for compression Large and have repetitive values Repetitive data tends to be clustered More clustered than in TPC-H tables Dimension tables are often too small for compression Large block size leads to greater compression Typical in data warehouses More rows available for compression within each block Materialized views can be compressed (and partitioned) Naturally sorted on creation due to GROUP BY Especially good for ROLLUP views and join views Tend to contain repetitive data 13

14 Compression of Individual Table Partitions Partition level Partitioning must be range or list (or composite) CREATE TABLE sales (sales_id NUMBER(8) : :,sales_date DATE) PARTITION BY RANGE(sales_date) (PARTITION sales_jan2003 VALUES LESS THAN (TO_DATE('02/01/2003','DD/MM/YYYY')) COMPRESS, PARTITION sales_feb2003 VALUES LESS THAN (TO_DATE('03/01/2003','DD/MM/YYYY')), PARTITION sales_mar2003 VALUES LESS THAN (TO_DATE('04/01/2003','DD/MM/YYYY')), PARTITION sales_apr2003 VALUES LESS THAN (TO_DATE('05/01/2003','DD/MM/YYYY'))); The first partition will be compressed Could consider compressing read only partitions of historical data 14

15 Effect of Partition Operations Consider individual partitions compressed as shown PARTITION p1 COMPRESS VALUES LESS THAN 100 PARTITION p2 COMPRESS VALUES LESS THAN 200 PARTITION p3 NOCOMPRESS VALUES LESS THAN 300 PARTITION p4 NOCOMPRESS VALUES LESS THAN 400 Splitting a compressed partition ALTER TABLE s1 SPLIT PARTITION p1 (50) INTO (PARTITION p1a, PARTITION p1b); Produces two new compressed partitions PARTITION p1acompress VALUES LESS THAN 50 PARTITION p1bcompress VALUES LESS THAN 100 PARTITION p2 COMPRESS VALUES LESS THAN 200 PARTITION p3 NOCOMPRESS VALUES LESS THAN 300 PARTITION p4 NOCOMPRESS VALUES LESS THAN

16 Effect of Partition Operations (contd) Effect of merging compressed partitions PARTITION p1acompress VALUES LESS THAN 50 PARTITION p1bcompress VALUES LESS THAN 100 PARTITION p2 COMPRESS VALUES LESS THAN 200 PARTITION p3 NOCOMPRESS VALUES LESS THAN 300 PARTITION p4 NOCOMPRESS VALUES LESS THAN 400 Merge of two compressed partitions ALTER TABLE s1 MERGE PARTITIONS p1b,p2 INTO PARTITION p1b_2; PARTITION p1a COMPRESS VALUES LESS THAN 50 PARTITION p1b_2 NOCOMPRESS VALUES LESS THAN 200 PARTITION p3 NOCOMPRESS VALUES LESS THAN 300 PARTITION p4 NOCOMPRESS VALUES LESS THAN 400 New partition p1b_2 is not compressed by default Same applies if any to be merged are initially uncompressed 16

17 Forcing Compression During Partition Maintenance Force compression of the new partition(s) after a split operation ALTER TABLE s1 SPLIT PARTITION p1 AT (50) INTO (PARTITION p1a COMPRESS,PARTITION p2a); Force compression of new partition after a merge operation ALTER TABLE s1 MERGE PARTITIONS p2,p3 INTO PARTITION p2_3 COMPRESS; Partitions may be empty or contain data during maintenance operations involving compression 17

18 Effect of Partitioned Bitmap Indexes Scenario : Table having no compressed partitions has bitmap locally partitioned indexes The presence of usable bitmap indexes will prevent the first operation that compresses a partition SQL> ALTER TABLE sales MOVE PARTITION p4 COMPRESS; ORA-14646: Specified alter table operation involving compression cannot be performed in the presence of usable bitmap indexes SQL> ALTER TABLE part2 SPLIT PARTITION p1a AT (25) INTO ( PARTITION p1c COMPRESS,PARTITION p1d); ORA-14646: Specified alter table operation involving compression cannot be performed in the presence of usable bitmap indexes 18

19 Compression of Partitions with Bitmap Indexes in Place Uncompressed partitioned table with bitmap index in 3 partitions CREATE TABLE emp_part PARTITION BY RANGE (deptno) (PARTITION p1 VALUES LESS THAN (11), PARTITION p2 VALUES LESS THAN (21), PARTITION p3 VALUES LESS THAN (31)) AS SELECT * FROM emp; CREATE BITMAP INDEX part$empno ON emp_part(empno) LOCAL; 19

20 Compression of Partitions with Bitmap Indexes in Place (continued) First compression operation requires the following 1. Mark bitmap indexes unusable (or drop them) ALTER INDEX part$empno UNUSABLE; 2. Compress the first (and any subsequent) partition as required ALTER TABLE emp_part MOVE PARTITION p1 COMPRESS; 3. Rebuild the bitmap indexes (or recreate them) ALTER INDEX part$empno REBUILD PARTITION p1; ALTER INDEX part$empno REBUILD PARTITION p2; ALTER INDEX part$empno REBUILD PARTITION p3; Each index partition must be individually rebuilt 20

21 Compression of Partitions with Bitmap Indexes in Place (continued) Oracle needs to know maximum records per block Correct mapping of bits to blocks can then be done On compression this value increases Oracle has to rebuild bitmaps to accommodate potentially larger number of values even if no data is present in the partition(s) Could result in larger bitmaps for uncompressed partitions Increase in size can be offset by the actual compression Once rebuilt, the indexes can cope with any compression Subsequent compression operations do not invalidate bitmap indexes Recommended to create each partitioned table with at least one compressed (dummy/empty?) partition Can be subsequently dropped Compression activity does not affect Btree usability 21

22 Table Level Compression for Partitioned Tables Compression can be the default for all partitions CREATE TABLE sales (sales_id NUMBER(8), : : sales_date DATE) COMPRESS PARTITION BY (sales_date)... Can still specify individual partitions to be NOCOMPRESS Default partition maintenance actions on compressed tables Splitting non-compressed partitions results in non-compressed partitions Merging non-compressed partitions results in a compressed partition Adding a partition will result in a new compressed partition Moving a partition does not alter its compression 22

23 Finding the Largest Tables Useful for finding candidates for compression SELECT owner,name,sum(gb),sum(pct) FROM (SELECT owner,name,to_char(gb,'999.99') gb,to_char((ratio_to_report(gb) OVER())*100,'999,999,999.99') pct FROM (SELECT owner,substr(segment_name,1,30) name,sum(bytes/(1024*1024*1024)) gb FROM dba_segments WHERE segment_type IN ('TABLE','TABLE PARTITION') GROUP BY owner,segment_name ) ) WHERE pct > 3 GROUP BY ROLLUP(owner,name) ORDER BY 3; 23

24 Finding the Largest Tables (contd) OWNER NAME SUM(GB) SUM(PCT) SH COSTS SH SALES SH SALES_HIST SH SYS IDL_UB2$ SYS SOURCE$ SYS

25 Sampling Data to Predict Compression 25

26 Compression Factor and Space Saving Compression Factor (CF) CF = non-compressed blocks compressed blocks * 100 Space Savings (SS) SS = non-compressed blocks - compressed blocks compressed blocks *

27 Predicting the Compression Factor CREATE OR REPLACE FUNCTION compression_ratio (tabname VARCHAR2) RETURN NUMBER IS pct NUMBER := ; -- sample percentage blkcnt NUMBER := 0; -- original block count (should be < 10K) blkcntc NUMBER; -- compressed block count BEGIN EXECUTE IMMEDIATE ' CREATE TABLE temp_uncompressed PCTFREE 0 AS SELECT * FROM ' tabname ' WHERE ROWNUM < 1'; WHILE ((pct < 100) AND (blkcnt < 1000)) LOOP -- until > 1000 blocks in sample EXECUTE IMMEDIATE 'TRUNCATE TABLE temp_uncompressed'; EXECUTE IMMEDIATE 'INSERT INTO temp_uncompressed SELECT * FROM ' tabname ' SAMPLE BLOCK (' pct ',10)'; EXECUTE IMMEDIATE 'SELECT COUNT(DISTINCT(dbms_rowid.rowid_block_number(rowid))) FROM temp_uncompressed' INTO blkcnt; pct := pct * 10; END LOOP; EXECUTE IMMEDIATE 'CREATE TABLE temp_compressed COMPRESS AS SELECT * FROM temp_uncompressed'; EXECUTE IMMEDIATE 'SELECT COUNT(DISTINCT(dbms_rowid.rowid_block_number(rowid))) FROM temp_compressed' INTO blkcntc; EXECUTE IMMEDIATE 'DROP TABLE temp_compressed'; EXECUTE IMMEDIATE 'DROP TABLE temp_uncompressed'; RETURN (blkcnt/blkcntc); END; / 27

28 Predicting the Compression Factor (continued) CREATE OR REPLACE PROCEDURE compress_test(p_comp VARCHAR2) IS comp_ratio NUMBER; BEGIN comp_ratio := compression_ratio(p_comp); dbms_output.put_line('compression factor for table ' p_comp ' is ' comp_ratio ); END; Run the compression test for the emp table EXEC compress_test('emp') Compression factor for table EMP is

29 Compression Test Clustered Data CREATE TABLE noclust (col1 VARCHAR2(1000)) COMPRESS; INSERT INTO noclust VALUES ('VV...VV'); INSERT INTO noclust VALUES ('WW...WW'); INSERT INTO noclust VALUES ('XX...XX'); INSERT INTO noclust VALUES ('YY...YY'); INSERT INTO noclust VALUES ('ZZ...ZZ'); INSERT INTO noclust VALUES ('VV...VV'); INSERT INTO noclust VALUES ('WW...WW'); INSERT INTO noclust VALUES ('XX...XX'); INSERT INTO noclust VALUES ('YY...YY'); INSERT INTO noclust VALUES ('ZZ...ZZ'); INSERT INTO noclust VALUES ('VV...VV'); : : : INSERT INTO noclust VALUES ('ZZ...ZZ'); Every value for column col1 is 390 bytes long CREATE TABLE clust (col1 VARCHAR2(1000)) COMPRESS; INSERT INTO clust VALUES ('VV...VV'); INSERT INTO clust VALUES ('VV...VV'); INSERT INTO clust VALUES ('VV...VV'); INSERT INTO clust VALUES ('VV...VV'); INSERT INTO clust VALUES ('VV...VV'); INSERT INTO clust VALUES ('WW...WW'); : : : INSERT INTO clust VALUES ('WW...WW'); INSERT INTO clust VALUES ('XX...XX'); : : : INSERT INTO clust VALUES ('YY...YY'); : : : INSERT INTO clust VALUES ('ZZ...ZZ'); Both tables have a total of 25 rows stored in blocks of size 2K So a maximum of four rows will fit in each block Both have same amount of repeated values but the clustering is different 29

30 Compression Test (continued) noclust - 4 rows per block. (7 blocks in total) The 5th row to be inserted must go in the next block as it contains different data header header header header vv vv zz zz yy yy xx xx ww ww vv vv zz zz yy yy xx xx ww ww vv vv zz zz yy yy xx xx ww ww vv vv header vv vv ww ww xx xx yy yy header zz zz clust - 20 rows per block. Rows 2,3,4,5 are duplicates of the first row in the block. Rows 7,8,9,10 are duplicates of the 6th row in the block, and this pattern is repeated. The residual space in the first block is used by the compressed data 30

31 Compression Test - Compression Factors Compression test routine is accurate due to sampling of actual data Make sure default tablespace is correctly set Temporary sample tables are physically built for the testing EXEC compress_test('clust') Compression factor for table CLUST is 3.5 EXEC compress_test('noclust') Compression factor for table NOCLUST is 1 31

32 Testing Compression : Using Repeatable sampling on Oracle9i Estimate the compression/decompression ratio for a table, abc 1. Make sampling repeatable: ALTER SESSION SET EVENTS '10193 trace name context forever, level 1'; This statement appears to have no effect in Oracle10g 2. Drop any previously created test tables DROP TABLE abc$test1; DROP TABLE abc$test2; 32

33 Testing Compression : Further Example (continued) 3. Create an empty compressed table: CREATE TABLE abc$test1 COMPRESS AS SELECT * FROM abc WHERE ROWNUM < 1; 4. Create an empty uncompressed table: CREATE TABLE abc$test2 NOCOMPRESS AS SELECT * FROM abc WHERE ROWNUM < 1; 5. Place the same test data in each of the test tables LOCK TABLE abc IN SHARE MODE; INSERT /*+ APPEND */ INTO abc$test1 SELECT * FROM abc SAMPLE BLOCK(x,y); INSERT /*+ APPEND */ INTO abc$test2 SELECT * FROM abc SAMPLE BLOCK(x,y); This example uses block level sampling 33

34 Testing Compression : Sampling Rows Tables can be sampled at row or block level Block level samples a random selection of whole blocks Row level (default) samples a random selection of rows SELECT * FROM emp SAMPLE (10); Selects a 10% sample of rows If repeated, a different sample will be taken Samples can be 'fixed' in Oracle10g using SEED SEED can can have integer values from Can also have higher numbers ending in '00' SELECT * FROM emp SAMPLE (10) SEED (1); Shows a 10% sample of rows If repeated, the exact same sample will be taken Also applies to block level sampling The sample set will change if DML is performed on the table 34

35 Pre-Sorting the Data for Compression 35

36 Sorting the Data for Compression Reorganize (pre-sort) rows in segments that will be compressed to cause repetitive data within blocks For multi-column tables, order the rows by the low cardinality column(s) CREATE TABLE emp_comp COMPRESS AS SELECT * FROM emp ORDER BY<some unselective column(s)>; For a single-column table, order the table rows by the column value Presort the data on a column which has : no. of distinct values ~ no. of blocks Information on column cardinality is shown in: ALL_TAB_COL_STATISTICS ALL_PART_COL_STATISTICS ALL_SUBPART_COL_STATISTICS 36

37 Sorting the Data for Compression Presort data on column having no. of distinct values ~ no. of blocks ~ SELECT * FROM large_emp; ( rows) EMPNO ENAME JOB ***** CLERK **** ANALYST ****** MANAGER : : : SELECT COUNT(DISTINCT job) FROM large_emp; 5 jobs SELECT COUNT(DISTINCT ename) FROM large_emp; 170 enames 37

38 Sorting the Data for Compression (continued) CREATE TABLE nocomp AS SELECT empno,ename,job FROM large_emp; Non-compressed table : Number of used blocks = 360 CREATE TABLE cjob COMPRESS AS SELECT empno,ename,job FROM large_emp ORDER BY job; Compressed table sorted on job : Number of used blocks = 243 CREATE TABLE cename COMPRESS AS SELECT empno,ename,job FROM large_emp ORDER BY ename; Compressed table sorted on ename : Number of used blocks = 172 Sorting on the job column is not the most effective 38

39 Behaviour of DML/DDL on Tables with Default Compression 39

40 Default Compressed Table Behaviour Updates, and conventional single and multi-row inserts are NOT compressed UPDATE Wholesale updates lead to large increases in storage (>250%) Performance impact on UPDATEs can be around 400% Rows are migrated to new blocks (default value of PCTFREE is 0) DELETE Performance impact of around 15% for compressed rows Creating a compressed table can take 50% longer Compressed tables cannot be modified in ORA cannot add column to object tables 40

41 Default Compressed Table Behaviour (continued) Operations which 'perform' compression CREATE TABLE... AS SELECT... ALTER TABLE... MOVE... INSERT /*+APPEND*/ (single threaded) INSERT /*+PARALLEL(sales,4)*/ Requires ALTER SESSION ENABLE PARALLEL DML; Both of the above inserts work with data from database tables and external tables SQL*Loader DIRECT = TRUE Various partition maintenance operations Can not be used for: LOB and VARRAY constructs Index organized tables Can use index compression on IOTs External tables, index clusters, hash clusters Bulk loads within PL/SQL Logical standby techniques 41

42 Compression Internals 42

43 Hexadecimal Dump of Compressed Data Symbol Table : 14 unique names 5 unique jobs Start of next block 43

44 Oracle Dump - Two Uncompressed Employee Rows ALTER SYSTEM DUMP DATAFILE 8 BLOCK 35; Creates a trace file in USER_DUMP_DEST... block_row_dump: tab 0, row tl: 41 fb: --H-FL-- lb: 0x2 cc: 8 col 0: [ 3] c col 1: [ 5] 43 4c b col 2: [ 7] 4d 41 4e col 3: [ 3] c2 4f 28 col 4: [ 7] 77 b col 5: [ 3] c col 6: *NULL* col 7: [ 2] c1 0b tab 0, row tl: 40 fb: --H-FL-- lb: 0x2 cc: 8 col 0: [ 3] c col 1: [ 5] f col 2: [ 7] 41 4e 41 4c col 3: [ 3] c2 4c 43 col 4: [ 7] 77 bb col 5: [ 2] c2 1f col 6: *NULL* col 7: [ 2] c

45 Oracle Dump of Table of empno,ename,job tl: 18 fb: --H-FL-- lb: 0x0 cc: 2 col 0: [ 9] e 54 col 1: [ 4] 4b 49 4e 47 bindmp: 00 0b 02 d e 54 cc 4b 49 4e 47 tab 0, row PRESIDENT KING tl: 9 fb: --H-FL-- lb: 0x0 cc: 2 col 0: [ 7] 41 4e 41 4c col 1: [ 4] 46 4f bindmp: 00 0a cc 46 4f FORD... tab 0, row tl: 10 fb: --H-FL-- lb: 0x0 cc: 2 col 0: [ 5] 43 4c b col 1: [ 5] 53 4d bindmp: 00 0b 02 0e cd 53 4d SMITH tab 0, row tl: 8 fb: --H-FL-- lb: 0x0 cc: 1 col 0: [ 5] 43 4c b bindmp: cd 43 4c b CLERK... tab 0, row tl: 10 fb: --H-FL-- lb: 0x0 cc: 1 col 0: [ 7] 41 4e 41 4c bindmp: cf 41 4e 41 4c tab 1, row tl: 9 fb: --H-FL-- lb: 0x0 cc: 3 col 0: [ 5] 43 4c b col 1: [ 5] 53 4d col 2: [ 3] c bindmp: 2c d cb c ANALYST 45

46 Oracle Avoids Unnecessary Compression Create two tables with repeating small values in one column CREATE TABLE tnocomp ( col1 VARCHAR2(1),col2 VARCHAR2(6)) PCTFREE 0; CREATE TABLE tcomp ( col1 VARCHAR2(1),col2 VARCHAR2(6)) COMPRESS; Insert data (320 rows) as follows COL1 COL A 1ZZZZZ 2A 2ZZZZZ 3A 3ZZZZZ 4A 4ZZZZZ 5A 5ZZZZZ 1A 6ZZZZZ 2A 7ZZZZZ... 4A 319ZZZ 5A 320ZZZ Values unique in col2 Values repeat in col1 every 5 rows 46

47 Evidence of Minimal Compression SELECT dbms_rowid.rowid_block_number(rowid) block,dbms_rowid.rowid_relative_fno(rowid) file,count(*) num_rows FROM &table_name GROUP BY dbms_rowid.rowid_block_number(rowid),dbms_rowid.rowid_relative_fno(rowid); tnocomp BLOCK FILE NUM_ROWS tcomp BLOCK FILE NUM_ROWS Evidence of compression in the 'compressed' table 47

48 Further Evidence of Compression ALTER SYSTEM DUMP DATAFILE 8 BLOCK 67; HEX(A) = 41 block_row_dump: tab 0, row tl: 5 fb: --H-FL-- lb: 0x0 cc: 1 col 0: [ 2] bindmp: 00 1b ca tab 0, row tl: 5 fb: --H-FL-- lb: 0x0 cc: 1 col 0: [ 2] bindmp: 00 1b ca tab 0, row tl: 5 fb: --H-FL-- lb: 0x0 cc: 1 col 0: [ 2] bindmp: 00 1b ca tab 0, row tl: 5 fb: --H-FL-- lb: 0x0 cc: 1 col 0: [ 2] bindmp: 00 1a ca tab 0, row tl: 5 fb: --H-FL-- lb: 0x0 cc: 1 col 0: [ 2] bindmp: ca tab 1, row tl: 12 fb: --H-FL-- lb: 0x0 cc: 2 col 0: [ 2] col 1: [ 6] bindmp: 2c c tab 1, row tl: 12 fb: --H-FL-- lb: 0x0 cc: 2 col 0: [ 2] col 1: [ 6] bindmp: 2c c Symbol Table (tab0) (1A,2A. 3A.4A.5A) 48

49 Compression not Performed on Unsuitable Data Both tables recreated with values in col1 now set to TO_CHAR(MOD(ROWNUM,50)) Much less repetition of values (only every 50 rows) allowing less compression tnocomp COL1 COL ZZZZZ 2 2ZZZZZ 3 3ZZZZZ 4 4ZZZZZ... 5ZZZZZ 49 49ZZZZ 50 50ZZZZ 1 51ZZZZ 2 52ZZZZ 3 53ZZZZ Oracle decides not to compress BLOCK FILE NUM_ROWS tcomp BLOCK FILE NUM_ROWS

50 Comparison of Heap and IOT Compression IOT = Index Organized table 50

51 Comparison of IOT and Heap Tables Tests constructed using a standard set of data in emptest Six columns with absence of nulls EMPNO ENAME JOB HIREDATE SAL DEPTNO KING PRESIDENT 17-NOV FORD ANALYST 03-DEC SCOTT ANALYST 09-DEC JONES MANAGER 02-APR BLAKE MANAGER 01-MAY CLARK MANAGER 09-JUN ALLEN SALESMAN 20-FEB TURNER SALESMAN 08-SEP MILLER CLERK 23-JAN WARD SALESMAN 22-FEB MARTIN SALESMAN 28-SEP ADAMS CLERK 12-JAN JAMES CLERK 03-DEC SMITH CLERK 17-DEC KING PRESIDENT 17-NOV FORD ANALYST 03-DEC SMITH CLERK 17-DEC IOT = Index Organized table 51

52 Creation of IOTs empi : Conventional IOT based on emptest data CREATE TABLE empi (empno,ename,job,hiredate,sal,deptno, CONSTRAINT pk_empi PRIMARY KEY(ename,job,hiredate,sal,deptno,empno)) ORGANIZATION INDEX PCTFREE 0 AS SELECT * FROM emptest empic : First five columns compressed CREATE TABLE empic (empno,ename,job,hiredate,sal,deptno, CONSTRAINT pk_empic PRIMARY KEY(ename,job,hiredate,sal,deptno,empno)) ORGANIZATION INDEX PCTFREE 0 COMPRESS 5 AS SELECT * FROM emptest 52

53 Test tables Four tables built having heap/iot structures and compressed/noncompressed data Unique index built on empno column in emph and emphc tables Table Table Compress Blocks Average Name Type row length emph Heap No emphc Heap Yes empi IOT No empic IOT Yes Average row length obtained from user_tables (avg_row_len) Compressed heap tables show no reduction in average row length 53

54 Compression Data Number of blocks in heap tables obtained using dbms_rowid SELECT COUNT(COUNT(*)) FROM &table_name GROUP BY dbms_rowid.rowid_block_number(rowid),dbms_rowid.rowid_relative_fno(rowid); Number of blocks in IOTs obtained from index validation VALIDATE INDEX &index_name; SELECT * FROM index_stats; Compressed IOTs have compression shown as DISABLED in user_tables, but ENABLED in user_indexes 54

55 Timings to Scan Tables SELECT COUNT(deptno) FROM <table_name>; Table Table Compress Elapsed Name Type Time (s) emph Heap No 0.70 emphc Heap Yes 0.47 empi IOT No 0.37 empic IOT Yes

56 Access Paths for Heap Table EXPLAIN PLAN FOR SELECT * FROM emph WHERE EMPNO > ; SELECT * FROM TABLE(dbms_xplan.display); PLAN_TABLE_OUTPUT Id Operation Name Rows Bytes Cost (%CPU) SELECT STATEMENT K 252 (2) 1 TABLE ACCESS BY INDEX ROWID EMPH K 252 (2) * 2 INDEX RANGE SCAN PK_EMPH (3) EXPLAIN PLAN FOR SELECT * FROM emph WHERE EMPNO > ; SELECT * FROM TABLE(dbms_xplan.display); PLAN_TABLE_OUTPUT Id Operation Name Rows Bytes Cost (%CPU) SELECT STATEMENT K 252 (5) * 1 TABLE ACCESS FULL EMPH K 252 (5)

57 Access Paths for Compressed Heap Table EXPLAIN PLAN FOR SELECT * FROM emphc WHERE EMPNO > ; SELECT * FROM TABLE(dbms_xplan.display); PLAN_TABLE_OUTPUT Id Operation Name Rows Bytes Cost (%CPU) SELECT STATEMENT K 86 (3) 1 TABLE ACCESS BY INDEX ROWID EMPHC K 86 (3) * 2 INDEX RANGE SCAN PK_EMPHC (3) EXPLAIN PLAN FOR SELECT * FROM emphc WHERE EMPNO > ; SELECT * FROM TABLE(dbms_xplan.display); PLAN_TABLE_OUTPUT Id Operation Name Rows Bytes Cost (%CPU) SELECT STATEMENT K 91 (11) * 1 TABLE ACCESS FULL EMPHC K 91 (11) Note that full table scan is (rightly) more likely to be performed 57

58 Access Path for IOTs EXPLAIN PLAN FOR SELECT * FROM empi WHERE EMPNO > ; SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY); PLAN_TABLE_OUTPUT Id Operation Name Rows Bytes Cost (%CPU) SELECT STATEMENT K 238 (1) * 1 INDEX FAST FULL SCAN PK_EMPI K 238 (1)

59 Updates to Heap and IOT Tables UPDATE <table_name> SET ename = 'XXXXXXX ; Lengthens each employee name by at least one character Table Table Compress Blocks Blocks Increase PCT Elapsed Name Type before after in blocks increase Time update update Emph Heap No ~ 10% 5 mins Emphc Heap Yes ~ 600% 15 mins Empi IOT No ~ 100% 4 mins Empic IOT Yes ~ 100% 12 mins Note the explosion in size of the compressed heap table 59

60 Advanced Compression in Oracle11g (for OLTP Operations) 60

61 Advanced Compression in Oracle 11g Conventional DML maintains the compression Inserted and updated rows remain compressed The compress activity is kept at a minimum Known as the Advanced Compression option Extra cost 61

62 Compressing for OLTP Operations Requires COMPATIBILITY = (or higher) CREATE TABLE t1... COMPRESS FOR ALL OPERATIONS ; Conventionally inserted rows stay uncompressed until PCTFREE is reached Mimimises compression operations Existing rows are not compressed Header Header Header Header Header Free space Free space Free space Free space Free space Conventional inserts are not compressed Block becomes full Rows are now compressed Transaction activity More uncompressed Inserts fill the block Rows are compressed again 62

63 OLTP Compression Some early results Table containing 3million parts records No compression 35M Compression for direct path operations 12M Compression for all operations 16M Avoid large scale (batch) updates on OLTP compressed tables Significant processing overheads 63

64 OLTP Compression Test Table containing sales records PROD_ID CUST_ID TIME_ID CHANNEL_ID PROMO_ID AMOUNT_SOLD ACCOUNT_TYPE JAN Minor account FEB Minor account NOV Minor account : : : : : : : Three tables created CREATE TABLE s_non_c PCTFREE 0 AS SELECT * FROM sales; Non-compressed table with fully packed blocks CREATE TABLE s_c COMPRESS AS SELECT * FROM sales; Compressed table for DSS operations CREATE TABLE s_c_all COMPRESS FOR ALL OPERATIONS AS SELECT * FROM sales; Compressed table for OLTP operations 64

65 OLTP Compression Test (continued) Update stress test update 94% of the rows No lengthening of any data UPDATE sales SET account_type = Major account WHERE prod_id > 13; Non-compressed table Compressed Table for DSS Compressed Table for OLTP Original size (blocks) Size after update (blocks) Elapsed time for update 39 secs 1:04 secs Test somewhat unfair on OLTP compression Update is large and batch orientated I/O subsystem was single disk But still interesting? 2:20:12 secs 65

66 Shrinking Unused Space in Oracle10g 66

67 Reclaiming Space in Oracle10g Unused space below the High Water Mark can be reclaimed online ALTER TABLE <table_name> SHRINK SPACE; Space caused by delete operations can be returned to free space Object must be in a locally managed tablespace with automatic segment space management (ASSM) Removes the need for Online Redefinition to reclaim space? Two-step operation Rows are moved to blocks available for insert Requires row movement to be enabled ALTER TABLE <table_name> ENABLE ROW MOVEMENT; High water mark (HWM) is repositioned Newly freed blocks are returned to free space if the COMPACT keyword is not used 67

68 Repositioning of HWM During shrinks HWM Allocated block above the high water mark HWM Free block Free block Free block Rows are physically moved to space in blocks on the freelist Shrinking causes ROWIDs to change Indexes (bitmap and btree) are updated accordingly Shrinking does not work with compressed tables 68

69 Shrinking Objects Why Shrink? To reclaim space To increase speed of full table scans To allow access to table during the necessary reorganisation The shrink operation can be terminated/interrupted at any time Can be continued at a later time from point of termination Objects that can be SHRINKed Tables Indexes Materialized views Materialized view logs Dependent objects may be shrunk when a table shrinks ALTER TABLE <table_name> SHRINK SPACE [CASCADE]; 69

70 Data Compression in Oracle Carl Dudley University of Wolverhampton, UK UKOUG SIG Director 70