XML Storage in DB2 Basics and Improvements Aarti Dua DB2 XML development July 2009

Transcription

1 IBM Software Group XML Storage in DB2 Basics and Improvements Aarti Dua DB2 XML development July 2009

2 Agenda Basics - XML Storage in DB2 Major storage enhancements Base Table Row Storage in DB2 9.5 Usage and tools Compression Storage improvement in DB2 9.7 XML Data Area Compression Usage and tools Upgrade considerations How all these improvements work together Summary/ Conclusion 2

3 XML Storage in DB2 The Basics Native XML Support in DB2 Introduced in DB2 9 DB2 can store, index and query XML Fixed XML schema per XML column is optional Indexes for specific attributes and elements XML can be used as the type of a column in a table: create table t (docid int, doc xml) Alter table t add column doc2 xml In DB2 9, XML support was available only on Unicode databases DB2 9.5 and onwards, Unicode restriction has been lifted 3

4 Storage Objects XML Data Area (XDA) Stores XML documents 1 XDA object per table Similar to DAT object for table data, XDA object stores XML data Base Table (DAT object) Stores XML descriptor Stored in the base table row in the XML column Contains information needed to locate the XML document in the XDA 4

5 Storage Objects Cont d Regions index System index 1 regions index per table with XML column Stores the RID of the XML documents in the XDA Index type XRGN in syscat.sysindexes syscat.sysindexes INDSCHEMA INDNAME TABNAME INDEXTYPE SYSIBM SQL TABLE1 XRGN 5

6 Storage Objects Cont d DAT Object (rel. Data) INX Object (Indexes) ID PR27 DOC (XML) Regions Index PR28 ACC XDA Object (XML Data) Like LOBs, XML data is stored separately from the base table. Unlike LOBs, XML data is buffered in the buffer pool. 6

7 Regions in a XML document XML documents can be larger than page size To store them XML documents are split into smaller regions A region Single-rooted subtree of nodes Region must fit on a single page Region can be smaller than a page 1 R R R1 7

8 Page Size for XML Regions Index Larger documents get split into regions Max. doc size: 2GB, spans many pages 4K Pages 8K Pages 32k Pages. Fewer regions per document are better for performance. Choose page size depending on document size, larger pages are better! 8

9 Agenda Basics - XML Storage in DB2 Major storage enhancements Base Table Row Storage in DB2 9.5 Usage and tools Compression Storage improvement in DB2 9.7 XDA Compression Usage and tools Upgrade considerations How all these improvements work together Summary/ Conclusion 9

10 Base Table Row Storage (aka. Inlining) Without use of Inlining All XML documents are stored in the XDA 2 step process to store & fetch docs Go through regions index Expensive! What is Inlining Store small XML documents in the XML column in the base table row Direct access to the XML documents No inserts, updates and fetches from the regions index No XDA storage needed for inlined documents Inlined documents can make use of row compression Better performance! 10

11 DB2 9.5: Base Table Inlining DAT Object (rel. Data) ID DOC (XML) PR27 PR28 ACC INX Object (Indexes) Regions Index Much smaller regions index, if most (or all!) documents are inlined. XDA Object (XML Data) 11

12 Inlining - Usage Create Table Stmt create table <table name> (<column name> xml INLINE LENGTH <integer>) Alter Table Stmt New XML column alter table <table name> add column <column name> xml INLINE LENGTH <integer> Modifying inline length alter table <table name> alter column <xml column name> set inline length <integer> 12

13 Small Example Create table sampletab ( docid int, xmldoc xml) Insert into sampletab values ( 1, <x><y>hello</y><z>world!</z></x> ) docid Xmldoc 1 XDA descriptor Regions Index Alter table sampletab alter column xmldoc set inline length 3000 Insert into sampletab values ( 2, <x><y>hello</y><z>world!</z></x> ) XDA <x><y>hello</y> ><z>world!</z> </x> 2 XDA descriptor <x><y>hello</y><z>world!</z></x> 13

14 How Can I Find the Inline Length of a Column SYSCAT.SYSCOLUMNS catalog Provides information on the columns of tables INLINE_LENGTH column shows Inline length specified for the column of the table 0 if no inline length is specified TABNAME COLNAME INLINE_LENGTH SAMPLETAB T1 XMLDOC XMLDOC

15 Inlining Cont d Conditions for document to be inlined Document must span only 1 region Bounded by inline size stated in the DDL Inline Length Restrictions Can increase the length but cannot decrease Inline lengths are based on Page size Row size Max inline length: Min inline length:

16 DB2 Info Center Refer to CREATE TABLE DDL Note: There is an additional 2 bytes of storage used by each row when VALUE COMPRESSION is active for that row. 16

17 Inlining Cont d Compress inlined XML documents Enable data row compression No additional setup is required to compress inlined XML! Side Effect of Inlining Increases base table row length Less rows per page Queries that only read relational columns may suffer 17

18 18 IBM Software Group DB2 Information Management Software Functions supporting Inlining Support is available in DB2 9.7 ADMIN_IS_INLINED Retrieves information about inline data for a XML, BLOB, CLOB or DBCLOB column Returns SMALLINT with 1 for inlined 0 for non-inlined NULL for NULL inputs SELECT PK, ADMIN_IS_INLINED(xml_doc1) as IS_INLINED from TAB1 PK IS_INLINED record(s) selected.

19 19 IBM Software Group DB2 Information Management Software Functions supporting Inlining Function ADMIN_EST_INLINE_LENGTH Support added in DB2 9.7 Returns Estimate of the inline length required to inline data Or negative value if the data cannot be inlined If the value is already inlined, Actual length of inline data Supports XML, CLOB, BLOB and DBCLOB columns SELECT PK, ADMIN_IS_INLINED(xml_doc1) as IS_INLINED, ADMIN_EST_INLINE_LENGTH (xml_doc1) as EST_INLINE_LENGTH from TAB1 PK IS_INLINED EST_INLINE_LENGTH record(s) selected.

20 Financial OLTP with FIXML - 1/3 Storage TPOX benchmark, Customer/Account documents: Order documents: Security documents: 4 20 kb 1 2 kb 3 10 kb 3.6M documents total, Page size: 16kb DAT pages XDA pages Regions index pages Total database size (GB) DB2 9 21, ,600 17, (100%) DB2 9.5 (more compact tree storage) 21, ,448 15, (70%) DB inlining 628, (66%) DB inlining + compression 259, (32%) 20

21 Performance Results 21

22 Agenda Basics - XML Storage in DB2 Major storage enhancements Base Table Row Storage in DB2 9.5 Usage and tools Compression Storage improvement in DB2 9.7 XDA Compression Usage and tools Upgrade considerations How all these improvements work together Summary/ Conclusion 22

23 XDA Compression Feature available in DB2 9.7 Extends DB2 row compression feature to the XDA Compress XML documents stored in the XDA In DB2 9.7 and onwards, when compression is enabled on a table Applies to the table (DAT) object *AND* the XDA object if it exists Separate XML dictionary XML dictionary is stored in the XDA object Data dictionary (as in DB2 9.0) is stored in the DAT object Steps leading to compression of documents in the XDA Enable compression on the table Automatic dictionary creation Triggers after 2MB of data in the XDA Compression of documents 23

24 Usage IBM Software Group DB2 Information Management Software Enable/ Disable Compression No new syntax introduced CREATE TABLE table1 ( docid int, docname varchar(20), doc xml) COMPRESS YES; ALTER TABLE table1 COMPRESS NO; Automatic dictionary creation (ADC) Triggers when data in the XDA crosses 2MB threshold Can be triggered during Insert/ Import LOAD Insert Redistribute (in DPF environment) Explicit ways to create XML Dictionary No new syntax introduced 24 REORG TABLE table1 LONGLOBDATA RESETDICTIONARY; DB2 LOAD from input.file of del REPLACE KEEPDICTIONARY INTO table1;

25 XDA Compression - Details Compression is applied to the documents if Compression is enabled on the table XML dictionary exists Decompression occurs upon access of the XML document When page is accessed in the bufferpool Range Partitioned tables Each partition has an XML dictionary Supported in DPF environment Same behavior as in Serial mode Decompressed when data is fetched from a DB2 node Decompressed data is transported between nodes 25

26 XDA Compression - Benefits Reduced storage needs for XML documents Up to 79% reduction is storage size of the XDA Better performance due to less I/O Import Runstats REORG LOAD Queries 26

27 XDA Compression Cont d Limitation XDA Compression is not supported on tables with XML column(s) that were created prior to the DB2 9.7 release To enable XDA compression on tables created using DB2 9 or DB2 9.5, migrate XML data via Online Table Move Export data Drop table Create table Import/ Load data Export data Truncate table Add XML Col Import/ Load data 27

28 XDA Compression ADMIN views and table functions ADMINTABINFO administrative view Associated table function ADMIN_GET_TAB_INFO_V97 Check whether XML dictionary exists Column XML_DICTIONARY_SIZE SELECT SUBSTR(TABSCHEMA, 1, 10) TBSCHEMA, SUBSTR(TABNAME, 1, 10) TBNAME, XML_DICTIONARY_SIZE, DICTIONARY_SIZE FROM SYSIBMADM.ADMINTABINFO WHERE TABNAME LIKE 'T%' TBSCHEMA TBNAME XML_DICTIONARY_SIZE DICTIONARY_SIZE ADUA T 0 0 ADUA T

29 XDA Compression ADMIN Views and Table Functions ADMINTABCOMPRESSINFO administrative view Associated table function ADMIN_GET_TAB_COMPRESS_INFO_V97 Information regarding compression of DATA and XML objects New column OBJECT_TYPE XML or DATA SELECT SUBSTR(TABSCHEMA, 1, 10) TBSCHEMA, SUBSTR(TABNAME, 1, 10) TBNAME, OBJECT_TYPE, COMPRESS_ATTR, DICT_BUILDER, ROWS_SAMPLED FROM SYSIBMADM.ADMINTABCOMPRESSINFO WHERE TABNAME LIKE 'T%' TBSCHEMA TBNAME OBJECT_TYPE COMPRESS_ATTR DICT_BUILDER ROWS_SAMPLED ADUA T DATA N NOT BUILT 0 ADUA T XML N NOT BUILT 0 ADUA T1 DATA Y REORG 1 ADUA T1 XML Y REORG 1 29

30 Should I enable Compression? How to estimate benefits of compression? Use ADMIN_GET_TAB_COMPRESS_INFO_V97 2 Modes of execution REPORT - default mode ESTIMATE SELECT SUBSTR(TABSCHEMA, 1, 10) TBSCHEMA, SUBSTR(TABNAME, 1, 10) TBNAME, OBJECT_TYPE, COMPRESS_ATTR, DICT_BUILDER, ROWS_SAMPLED, PAGES_SAVED_PERCENT FROM TABLE (SYSPROC.ADMIN_GET_TAB_COMPRESS_INFO_V97('ADUA', 'T', 'ESTIMATE')) as T; TBSCHEMA TBNAME OBJECT_TYPE COMPRESS_ATTR DICT_BUILDER ROWS_SAMPLED PAGES_SAVED_ PERCENT ADUA T DATA N TABLE FUNCTION 1 63 ADUA T XML N TABLE FUNCTION

31 Agenda Basics - XML Storage in DB2 Major storage enhancements Base Table Row Storage in DB2 9.5 Usage and tools Compression Storage improvement in DB2 9.7 XDA Compression Usage and tools Upgrade considerations How all these improvements work together Summary/ Conclusion 31

32 Upgrade Considerations DB2 9 DB2 9.5 Compact storage Seen up to 54% reduction in XML storage DB2 9.5 DB2 9.7 Change in the storage format of the XML documents Storage format of DB2 9 and DB2 9.5 Type-1 XML record format Storage format of DB2 9.7 Type-2 XML record format Some DB2 9.7 features require Type-2 XML record format XDA Compression Redistribute 32

33 Upgrade Considerations Cont d Change in XML Record Format A table cannot contain XML documents in both formats Without any data migration In DB2 9.7, a table with XML documents in Type-1 record format will continue to store documents in Type-1 format How to determine the record format of a table ADMINTABINFO administrative view & ADMIN_GET_TAB_INFO_V97 table function Column XML_RECORD_TYPE Values 1 if Type-1 XML record format 2 if Type-2 XML record format NULL if table has no XML columns 33

34 Upgrade Considerations Cont d SELECT SUBSTR(TABSCHEMA, 1, 10) TBSCHEMA, SUBSTR(TABNAME, 1, 10) TBNAME, XML_RECORD_TYPE FROM SYSIBMADM.ADMINTABINFO WHERE TABNAME LIKE 'T%' TBSCHEMA ADUA ADUA ADUA TBNAME T T1 T2 XML_RECORD_TYPE

35 Upgrade Considerations Cont d How to migrate XML data into Type-2 storage format Online Table Move Export data Drop table Create table Import/ Load data Export data Truncate table Add XML Col Import/ Load data 35

36 Putting it all together XML Storage feature Enhanced storage Base Table Row Storage XDA Compression When can it be used In DB By default In DB When workload involves small XML documents - Need to enable the table In DB When workload involves large XML documents - Need to enable the table - Requires Type-2 XML record format 36

37 Conclusion To remember - XML storage format has changed in DB2 9.7 Depending on the workload, make an informed decision on how to store XML in DB2 Compacted storage introduced in DB2 9.5 Base Table Row Storage XDA Compression The right XML storage option can make a huge impact on Performance! Performance! and Performance! 37

38 Acknowledgements Thanks to the following people for reviewing the document and for their valuable feedback Matthias Nicola Christina Lee Susanne Englert Susan Malaika 38

39 Further reading XML storage 9r7/index.jsp?topic=/com.ibm.db2.luw.xml.doc/doc/ c html Data Compression 9r7/index.jsp?topic=/com.ibm.db2.luw.admin.perf.d oc/doc/c html 39