XML Index Overview for DB2 9 for z/os

Transcription

1 DB2 z/os XML Core Development & Solutions XML Index Overview for DB2 9 for z/os Rick Chang, Xiaopeng Xiong 10/5/ IBM Corporation

2 Agenda 1. XML Index Creation by Rick Chang 1. PureXML Basics. 2. Creating XML Index. 3. XMLPattern Expressions. 4. UNIQUE keyword 5. Recommendations 2. XML Index Exploitation by Xiaopeng Xiong 1. XML Index Exploitation For XMLEXISTS and XMLTABLE 2. Index Matching Rules and Index Recommendations 3. New Access Methods 4. On-going XML Index Enhancements 2

3 PureXML Basics CREATE TABLE PurchaseOrders ( ORD_NO INTEGER NOT NULL, XMLPO XML ); XML Table DOCID MIN_NODEID XMLDATA Base Table ORD_NO XMLPO DOCID DOCID When a table with XML columns is created or ALTER a table to add XML columns, DB2 implicitly creates the following objects: 1. A table space and table for each XML column. 2. A column, called DB2_GENERATED_DOCID_F OR_XML(or DOCID in short) in the base table to hold a unique document identifier. 3. An XML indicator column in the base table for each XML column. 4. An index on the document identifier. 5. An index on each XML table that DB2 uses to maintain document order. NODEID 3

4 PureXML Basics (cont) INSERT INTO PurchaseOrders (ORD_NO, XMLPO) VALUES(100, XMLPARSE(DOCUMENT <?xml version="1.0"?><purchaseorder ')); XML Table DOCID MIN_NODEID XMLDATA Base Table ID XMLPO DOCID xxx yyy xxx xxx NODEID DOCID When data is inserted: 1. An unique document identifier will be generated and insert into base table. 2. The XML data will be inserted into XML table. 4

5 XML Index An XML index is created to improve the efficiency of queries on XML documents. An XML index uses an XMLPattern expression to index paths and values in XML document. An XML index provides access to nodes within the document by creating index key based on XMLPattern expression. An XMLEXISTS predicate or XMLTABLE function can be indexable if there is an XML index with an XMLPattern that matches the XPath expression in the XMLEXISTS predicate or XMLTABLE function. 5

6 Creating XML Index XML column CREATE INDEX IDX1 ON PurchaseOrders( XMLPO ) GENERATE KEYS USING XMLPATTERN '/purchaseorder/items/item/desc' AS SQL VARCHAR(100) ; Keywords to specify XML values index SQL data-type 6

7 Restrictions XMLPATTERN keyword is required The column specified in the ON clause must be a column of XML data type Only one column can be specified in the ON clause ASCending/DESCending is not allowed in the ON clause XML values index cannot be CLUSTER PARTITION BY PADDED PARTITIONED 7

8 XML Index XML index indexes values of elements or attributes inside a document. Index entries include key value, DocID and NodeID. Support string (VARCHAR) and numeric (DECFLOAT) key type CREATE INDEX IDX1 ON PurchaseOrders(XMLPO) Generate Keys Using XMLPATTERN /purchaseorder/items/item/desc as SQL VARCHAR(100); <?xml version="1.0"?> <purchaseorder orderdate= "> <shipto country="us"> <name>alice Smith</name>... </shipto> <billto country="us"> <name>robert Smith</name>... </billto> <comment>hurry, my lawn is going wild!</comment> <items> <item partnum="872-aa"> <desc>lawnmower</desc> <quantity>1</quantity> <USPrice>148.95</USPrice> <comment>confirm this is electric</comment> </item> <item partnum="926-aa"> <desc>baby Monitor</desc> <quantity>1</quantity> <USPrice>39.98</USPrice> <shipdate> </shipdate> </item> </items> </purchaseorder> 8

9 More on XML index The number of keys for each document (each base row) depends on the document and XMLPattern. May have zero or more index entries per document (each base row). For a numeric index, if a string from a document cannot be converted to a number, it is ignored. <a><b>x</b></a>, XMLPattern /a/b as SQL DECFLOAT. No entry will be created in the index. <a><b>x</b><b>5</b></a>, XMLPattern /a/b as SQL DECFLOAT. Only one entry 5 will be created in the index. For a string (VARCHAR(n)) index, if a key value is longer than the limit, INSERT or CREATE INDEX will fail. Guarantee that the index entry will be created for string index. To test for the existence of a node. Create string index. Example: XMLEXISTS( /a/b PASSING xmlcol) 9

10 XMLPattern Expressions Supports subset of XPath expression with following limitations Maximum 4000 bytes after converted to UTF-8. XPath axis cannot be more than 50 levels. No predicates. No comment and no PI support. May include optional namespace declaration. Example: CREATE INDEX CUST_PHONE_IDX on Customer(Info) GENERATE KEY USING XMLPATTERN AS SQL VARCHAR(12) CREATE INDEX CUST_PHONE_IDX on Customer(Info) GENERATE KEY USING XMLPATTERN /child::customerinfo/child::phone/attribute::type AS SQL VARCHAR(12) CREATE INDEX CUST_PHONE_IDX2 on Customer(Info) GENERATE KEY USING XMLPATTERN declare default element namespace AS SQL DECFLOAT CREATE INDEX CUST_PHONE_IDX2 on Customer(Info) GENERATE KEY USING XMLPATTERN declare default element namespace /descendant-or-self::node()/attribute::type AS SQL DECFLOAT 10

11 SQL Types AS SQL DECFLOAT DECFLOAT (34) AS SQL VARCHAR(n) 1 <= n <= 1000 Can create multiple indices to index the XML value in multiple SQL types. Example: <customerinfo Cid= 362 > </customerinfo> CREATE INDEX CUST_IDX_CHAR on Customer(Info) GENERATE KEY USING XMLPATTERN /customerinfo/@cid AS SQL VARCHAR(4) CREATE INDEX CUST_IDX_NUM on Customer(Info) GENERATE KEY USING XMLPATTERN /customerinfo/@cid AS SQL DECFLOAT SELECT Info FROM Customer WHERE XMLEXISTS( /customerinfo[@cid > 82 ] PASSING Info); CUST_IDX_CHAR index will be used. SELECT Info FROM Customer WHERE XMLEXISTS( /customerinfo[@cid > 82 ] PASSING Info); CUST_IDX_NUM index will be used. 11

12 UNIQUE keyword Enforces uniqueness of the key values after casting to SQL type. For DECFLOAT SQL type, if key can not be cast to DECFLOAT, no enforcement for uniqueness. Enforces uniqueness across all documents in an XML column. 12

13 Recommendations Similar to Relational Indices When creating index on a table with large amount of data, it is recommended to create the XML index with DEFER YES option, rebuild index by using DB2 REBUILD INDEX utility. Check the Real Time Statistics (RTS) for activity on the indexspace for the XML index. Drop the index if there is no activity on the index. Use separate buffer pool for XML indices. It is easier to monitor the XML index activity. 13

14 XML Index Create or Rebuild Create Elapsed Create CPU Rebuild Elapsed Rebuild CPU Time in second //e //e' /a/b/@c' /a/b/@c /a/b/f/g /a/b/f/g' 14

15 Insert XML with indices Insert Elapsed and CPU 160% 140% 120% 100% 80% 60% 40% 20% 0% 138% 125% 111% 116% 100% 100% Elapsed CPU XML w/1 index w/ 2 indexes 15

16 XML Index Exploitation 16

17 What You Have Learned For XML Index Creation 1. Understand base table, XML table, DOCID index, NODEID index and user created XML index. 2. Syntax, keywords and restrictions on the creation of XML index. 3. Understand XMLPattern. 4. SQL type for index key and different behavior on key generation. 5. What is UNIQUE keyword enforcing. 6. Recommendations and the performance charts. 17

18 XML Index Exploitation For DB XML Index Exploitation For XMLEXISTS and XMLTABLE 2. Index Matching Rules and Index Recommendations 3. New Access Methods For XML Index 4. On-going XML Index Enhancements (optional) 18

19 Index Exploitation Determine the best eligible XML index Get DOCID list of qualified key entries via the XML index Convert DOCID list to RID list via the DOCID index Reevaluate XPath to ensure correctness Access XML documents via the NodeID index Access the base table rows 19

20 Index Exploitation DOCID Index Information Management CREATE INDEX IDX1 ON PurchaseOrders(XMLPO) GENERATE KEY USING XMLPATTERN /purchaseorder/items/item/desc AS SQL VARCHAR(20) Exploit IDX1, get DOCIDs with dup removed (E.g., 1001, 1002) 1 Via DocID index, get base table RID and access base table rows (E.g.,: the first two rows) /purchaseorder/items/item/desc KeyValue DOCID Air Conditioner 1003 Baby Monitor 1001 Baby Monitor 1002 Lawnmower IDX1 PurchaseOrders (Base Table) DOCID ORD_NO SELECT ORD_NO FROM PurchaseOrders WHERE XMLEXISTS( /purchaseorder/items/item[desc= Baby Monitor ] PASSING XMLPO); NodeID Index Via NodeID index, access XMLPO. Re-evaluate XPath (required) 3 XMLPO XML desc XML desc XML desc (Internal XML Table) DOCID XMLPO <purchaseorder orderdate= ">... <items> <item partnum="872-aa"> <desc>lawnmower</desc> <quantity>1</quantity> </item> <item partnum="926-aa"> <desc>baby Monitor</desc> <quantity>1</quantity> </item> </items> </purchaseorder> <purchaseorder orderdate= ">... <items> <item partnum= 926-AA"> <desc>baby Monitor</desc> <quantity>1</quantity> </item> <items> </purchaseorder> <purchaseorder orderdate= ">... <items> <item partnum= 957-BB"> <desc>air Conditioner</desc> <quantity>2</quantity> </item> <items> </purchaseorder>

21 Points To Note XMLEXISTS and XMLTable are eligible for exploiting XML index Not eligible for XMLQUERY XML index pages and base table rows are NOT locked during index exploitation Provide optimal concurrency and throughput XPath is re-evaluated after index exploitation, for two reasons Index not exact match, may get more docs back Possible updates during index exploitation 21

22 Exploring XML Index For XMLTABLE Row XPath expression of XMLTABLE can explore XML index The index matching criteria of XMLTABLE is the same as that of XMLEXISTS (info in the following slides) SELECT X.* FROM PurchaseOrders, XMLTABLE( /purchaseorder/items/item[desc= Baby Monitor ] PASSING PurchaseOrders.XMLPO COLUMNS quantity INTEGER, desc VARCHAR(100)) X; If the following index exists, DB2 can explore the index for the above query XMLPATTERN /purchareorder/items/item/desc AS SQL VARCHAR(100); 22

23 Index Match: Exact Match General rule: index contains all data that the querying XPath needs Exact Match: the XML index pattern is the same as the XPath in XMLEXISTS/XMLTABLE (including the ending part in a predicate) Exact Matches: XML Index Pattern /a/b/c Data Type of Index varchar XPath in XMLEXISTS/XMLTABLE /a/b/c /a/b[c < abc ] Index Access An XML index of SQL VARCHAR type can be used for XPath containing predicates of string comparison, or existential predicates An XML index of SQL DECFLOAT type can be used for XPath containing predicates of numeric comparison, //a/b /a/b/c varchar decfloat //a/b //a[b = abc ] /a/b[c < 5.2] /a/b/c /a/b[c < abc ] 23

24 Index Match: Partial Match General rule: index contains all data that the querying XPath needs Partial Match: the XML index pattern is less restrictive than the querying XPath, i.e., the index contains a super set of data than the querying XPath needs Partial Matches: XML Index Pattern /a/b/c Data Type of Index varchar XPath in XMLEXISTS/XMLTABLE /a[b/c/d] /a/b[c= abc ]//d Index Access An XML index of SQL VARCHAR type can be used for XPath containing predicates of string comparison, or existential predicates /a/*/c //a/b varchar varchar /a/b/c /a/b[c< abc ]//d /s/a[b= abc ]/c An XML index of SQL DECFLOAT type can be used for XPath containing predicates of numeric comparison //a//b 24

25 Index Match: ANDing and ORing The querying XPath may consist of multiple segments which are ANDed or ORed together If multiple XPath segments can use XML indexes, ANDing or ORing the DOCID lists of individual segments to produce a final DOCID list For AND operator, only one operand needs to be qualified for index exploitation ANDing and ORing XML Index Pattern /a/b/c, /a/b/d /a/b/c, /a/b/d decfloat decfloat Data Type of Index XPath in XMLEXISTS/XMLTABLE /a/b[c = 5 and d > 8] /a/b[c = 5 or d > 8]/e /a/b[c = 5 or d > abc ] Index Access For OR operator, either both operand must or no operand can be qualified for index exploitation 25

26 Determine The Best Index Multiple possible matching methods may apply when there are multiple indexes created on the same XML column. DB2 uses the following heuristic rules to determine the best index to use: - Matching index type - Exact match is better than partial match - For partial match, matching with more steps is better than matching with less steps - For partial match, matching without recursive steps (descendant axis, descendantor-self axis) is better than matching with recursive steps. Recursive steps may return many unnecessary index entries Example: XML indexes: idx1 /a/b of varchar type idx2 /a/b/c/d of varchar type idx3 //a of varchar type XPath expression: /a/b/c[d = abc ] DB2 uses the indexes in the following preference order: idx2, idx1, idx3 26

27 Recommendations For XML Pattern of Index create table customer(info XML); Assume we expect predicates on phone <customerinfo cid= 1004 > <name>matt Foreman</name> <phone> </phone> </customerinfo> Option 1: create index idx1 on customer(info) generate key using xmlpattern '/customerinfo/phone' as SQL varchar(40); Option 2: create index idx2 on customer(info) generate key using xmlpattern '//phone' as SQL varchar(40); Option 3: create index idx3 on customer(info) generate key using xmlpattern '/customerinfo/*' as SQL varchar(40); XML Index should contain query predicates Specify full path for index XML patterns, avoid wild card, or descendant axis Wildcard * and // are only used for a real reason. * for generic search need // for recursive documents or nodes appearing at different paths or levels. 27

28 XML Index Related Access Methods Access Methods Description Example DocScan R Directly scan XML documents and evaluate XPath /purchaseorder/items/item[usprice > 100] with no eligible index DOCID list access DX DOCID ANDing/ORing DI/DU Unique DOCID list from an XML index, then access the base table and XML table Intersect or union (unique) DOCID lists from XML indexes, then access the base table and XML table /purchaseorder/items/item[usprice > 100] with index on /purchaseorder/items/item/usprice as SQL DECFLOAT /purchaseorder/items/item[usprice > 100 and quantity > 5] with indexes on: /purchaseorder/items/item/usprice as SQL DECFLOAT and /purchaseorder/items/item/quantity as SQL DECFLOAT 28

29 EXPLAIN The Access Methods TABLE DDL CREATE TABLE PurchaseOrders (ORD_NO INT, XMLPO XML); INDEX DDL CREATE INDEX IDX1 ON PurchaseOrders(XMLPO) GENERATE KEY USING XMLPATTERN '/purchaseorder/items/item/usprice' AS SQL DECFLOAT; EXPLAIN QUERY CREATE INDEX IDX2 ON PurchaseOrders(XMLPO) GENERATE KEY USING XMLPATTERN '/purchaseorder/items/item/quantity' AS SQL DECFLOAT; EXPLAIN ALL SET QUERYNO = 1 FOR SELECT XMLQUERY('/purchaseOrder/items/item[USPrice > 100 or quantity > 5]' PASSING XMLPO) FROM PurchaseOrders WHERE XMLEXISTS('/purchaseOrder/items/item[USprice > 100 or quantity > 3]' PASSING XMLPO); EXPLAIN TABLE PLANNO ACCESSTYPE ACCESSNAME MATCHCOLS MIXOPSEQ _ 1 M 0 0 2_ 1 DX IDX _ 1 DX IDX _ 1 DU

30 What Have You Learned For Index Exploitation 1. How are indexes used in DB2 XMLEXISTS predicate XMLTABLE function 2. How does DB2 determine the best index for a particular query Exact Match Partial Match ANDing and ORing 3. Recommendations for creating good XML index 4. Access methods using XML index 30

31 Enhancements For XML Index In DB2 9 Support XML index exploitation for XMLEXISTS predicate with RHS being a string constant with length longer than the index key length, or a variable bound to a host variable, parameter marker, XML-typed item sequence, scalar fullselect subquery result, etc. PK55783 / PK81260 (closed apars, UK46726 / UK50085) Examples CREATE INDEX I1 ON PurchaseOrders(XMLPO) GENERATE KEY USING XMLPATTERN '/purchaseorder/items/item/desc' AS SQL VARCHAR(15); SELECT ORD_NO FROM PurchaseOrders WHERE XMLEXISTS('/purchaseOrder/items/item[desc=$x]' PASSING XMLPO, XMLQUERY('//text()' PASSING XMLPARSE('<z>Baby Monitor</z>')) AS "x"); RHS is a var bound to a sequence of items SELECT ORD_NO RHS is a string constant FROM PurchaseOrders longer than varchar(15) WHERE XMLEXISTS('/purchaseOrder/items/item[desc= This product is for business use only"]' PASSING XMLPO); 31

32 Enhancements For XML Index In DB2 9 (cont.) Support XML index containing certain XPath functions fn:upper-case(), fn:substring(), fn:starts-with(), fn:not(), fn:exists() PK80732 / PK80735 Examples CREATE INDEX IDX1 ON PurchaseOrders(XMLPO) Generate Keys Using XMLpattern '/purchaseorder/items/item/fn:exists(shipdate)' AS SQL VARCHAR(1); SELECT ORD_NO FROM PurchaseOrders WHERE XMLEXISTS('/purchaseOrder/items/item[fn:not(shipDate)] PASSING XMLPO); CREATE INDEX IDX2 ON PurchaseOrders(XMLPO) Generate Keys Using XMLpattern '/purchaseorder/items/item/desc/fn:upper-case(.)' AS SQL VARCHAR(20); SELECT ORD_NO FROM PurchaseOrders WHERE XMLEXISTS('/purchaseOrder/items/item[fn:upper-case(desc)= BABY MONITOR ] PASSING XMLPO); 32

33 Enhancements For XML Index In DB2 9 (cont. 2) The XPath in XMLEXISTS is re-evaluated after XML index matches For two reasons: not exact match, and possible update Evaluating XPath using scan is quite CPU intensive Under certain condition, re-evaluation can be avoided and performance improved The XPath in the predicates matches exactly with the XPath for the index No update to the XML document since the start of the index scan Avoidance of re-evaluation for XMLEXISTS after exact XML index matching PK

34 Q & A 34