On the Use of Query-driven XML Auto-Indexing

Transcription

1 On the Use of Query-driven XML Auto-Indexing Karsten Schmidt and Theo Härder SMDB'10 (ICDE), Long Beach March, 1

2 Motivation Self-Tuning '10 The last 10+ years Index tuning What-if Wizards, Guides, Druids Monitoring Workload analysis Static resources Single-user optimization Today: dynamic environments (queries, data, resources) Self-tuning has to be done Permanently and Online 1

3 Motivation Index Tuning Which column(s) to index? ID Name Age DepID Location 1 Norman 28 12A North 2 Charles West 2

4 Motivation Index Tuning Which column(s) to index? PK? ID Name Age DepID Location 1 Norman 28 12A North 2 Charles West 2

5 Motivation Index Tuning Which column(s) to index? ID Name Age DepID Location 1 Norman 28 12A North 2 Charles West query-supporting candidate? 2

6 Motivation Index Tuning Which column(s) to index? ID Name Age DepID Location 1 Norman 28 12A North 2 Charles West multiple candidates? 2

7 Motivation Index Tuning Which column(s) to index? ID Name Age DepID Location 1 Norman 28 12A North 2 Charles West multi-column index or multiple indexes? 2

8 Motivation Index Tuning Which column(s) to index? ID 1 2 ID ID Name Age DepID Location Name Age DepID Location Name DepID Norman Age28 12A Location North 1 columnsnorth and 1ISP: 10 tables Normaneach 28having 8 12A Norman 28 index 2 3 columns Charles 58 12A 98 North West max per ~4,000 indexes 2 Charles West Charles West redundancy and maintenance! 2

9 Motivation Index Tuning Which column(s) to index? ID Name Age DepID Location ID Name Age DepID Location ID 1 Name Age28 DepID Norman 12A Location North ISP: 10 tables each having 8 columns and 1 Norman 28 12A North 1 max Norman 28 index 2 3 columns Charles 58 12A 98 North West per ~4,000 indexes 2 Charles West 2 Charles West XML specifics on top of the ISP Semi-structured and schema changes Indexing may focus on structure and/or content Flexible path expressions (axes, wildcard, name) containment problem and generalized indexes descendant axis is evil, but wildcards too Node types (elements, attributes, text, ) 2

10 Outline XML Index Types Statistics for Cost-based Index Selection Index Configuration Management Evaluation Summary & Outlook

11 Outline XML Index Types Statistics for Cost-based Index Selection Index Configuration Management Evaluation Summary & Outlook

12 XML Index Types - Storage XML Mapping Native No Shredding or Blob Fundamentals Node labeling B-tree 4

13 XML Index Types - Storage bib XML Mapping Elementless publication book year 1994 id book title 1 book author TCP/IP last Stevens Structure price first content W. 5

14 XML Index Types - Storage 1 XML Mapping Elementless year id book title book author TCP/IP publication Structure book price last Stevens bib first content W

15 XML Index Types - Storage 1 XML Mapping Elementless year id book title book author TCP/IP publication Structure book price last Stevens bib first content W

16 XML Index Types - Storage 1 XML Mapping Elementless id year book title author TCP/IP publication Structure book price last Stevens Path Synopsis book bib first content W bib 2 publication 3 book year 4 id 5 12 journal type title author last 9 first 10 price 11 publisher 13 name 14 loc 15 paper 16 name title

17 XML Index Types - Storage XML Mapping Elementless Structure TCP/IP Stevens Path Synopsis + B-tree layout W bib publication 3 book year 4 id 5 12 journal type title author last 9 first 10 price 11 publisher 13 name 14 loc paper 16 name title document index TCP/IP Stevens W DeweyID content content document container PCR 5

18 XML Index Types - Storage XML Mapping Elementless TCP/IP Primary target: space efficiency65.95 content Secondary: more indexing options for free! Stevens W. Path Synopsis + B-tree layout bib publication 3 book year 4 id 5 12 journal type title author last 9 first 10 price 11 publisher 13 name 14 loc paper 16 name title TCP/IP Stevens W DeweyID document index content document container PCR 5

19 XML Index Types Content Element 6

20 XML Index Types Content Element Path Content And Structure Path: /bib//title PCRs: 5,11,33 Path: /bib//title PCRs: 5,11,33 6

21 XML Index Types Content Easy to define Generic Large Maintenance costs Path /bib//title PCRs: 5,11,33 Specific Clustering Small Hard to define Path: Element Easy to define Generic False positive filtering Large Maintenance costs Content And Structure Path: /bib//title PCRs: 5,11,33 Specific Clustering Medium size No document order Type support Hard to define 6

22 XML Index Types Usage Sample Simplified Query Graph Model for XMark 01 Query: let $auction := doc("auction.xml") return for $b in = "person0"] return $b/name/text() 7

23 XML Index Types Usage Sample Simplified Query Graph Model for XMark 01 Query: let $auction := doc("auction.xml") return for $b in = "person0"] return $b/name/text() 7

24 XML Index Types Usage Sample Simplified Query Graph Model for XMark 01 Query: let $auction := doc("auction.xml") return for $b in = "person0"] return $b/name/text() 7

25 XML Index Types Usage Sample Simplified Query Graph Model for XMark 01 Query: let $auction := doc("auction.xml") return for $b in = "person0"] return $b/name/text() 7

26 XML Index Types Usage Sample Simplified Query Graph Model for XMark 01 Query: let $auction := doc("auction.xml") return for $b in = "person0"] return $b/name/text() 7

27 XML Index Types Usage Sample Simplified Query Graph Model for XMark 01 Query: let $auction := doc("auction.xml") return for $b in = "person0"] return $b/name/text() 7

28 XML Index Types Usage Sample Simplified Query Graph Model for XMark 01 Query: let $auction := doc("auction.xml") return for $b in = "person0"] return $b/name/text() Besides element and content index More than 280 path or CAS indexes possible by varying depth, axes ( /, // ), clustering, and wildcards 7

29 Outline XML Index Types Statistics for Cost-based Index Selection Index Configuration Management Evaluation Summary & Outlook

30 Statistics Extended Path Synopsis Index Statistics 1 bib 2 publication 3 book year 4 id 5 type title author 6 12 journal 7 8 last 9 price publisher 11 first 13 name loc 15 paper 16 name 17 pages, size title 18 Each node is extended: 12 journal Instance counter o content length IUD counter Overhead: h Extended PS typically <0.1% of XML document No optimization yet (e.g., recursion) Processing overhead less than 5-6% Gathered during storage: B-tree height # leave pages Cardinality Index size 9

31 Statistics Cost Estimation for Index Candidates 1.Evaluate Index Expression on Path Synopsis PCR set 2.For each PCR add node's statistics (cardinality, width, IUD counter) 3.Index type-dependent estimation of height, no. of leaves, and size Estimation Accuracy 10

32 Outline XML Index Types Statistics for Cost-based Index Selection Index Configuration Management Evaluation Summary & Outlook

33 Index Configuration Management Integrated into XTC backend Asynchronous jobs for Candidate search Index building Flexible configuration Type support Pruning thresholds Size limits Job schedule 12

34 Index Configuration Management 1. Record query processing costs 2. If cost > threshold do Auto Indexing 3. Feedback Auto Indexing costs Auto Indexing 1. Traverse query plan bottom-up (access operators) 2. Generate index candidates 3. Rerun optimization including candidates 4. Analyze plan(s) for selected candidates 5. Update candidate set Cost/benefit calculation schedule index materialization jobs 13

35 Candidate Generation Auto Indexing Traverse Query: let $doc:=doc("sample.xml") return for $book in return $book/title Candidates: QGM representation 14

36 Candidate Generation Auto Indexing Traverse Query: let $doc:=doc("sample.xml") return for $book in return $book/title Candidates: QGM representation 1. // 14

37 Candidate Generation Auto Indexing Traverse Query: let $doc:=doc("sample.xml") return for $book in return $book/title Candidates: QGM representation 1. // 2. book 14

38 Candidate Generation Auto Indexing Traverse Query: let $doc:=doc("sample.xml") return for $book in return $book/title Candidates: QGM representation 1. // 2. book 14

39 Candidate Generation Auto Indexing Traverse Query: let $doc:=doc("sample.xml") return for $book in return $book/title Candidates: QGM representation 1. // 2. book 4. book1 14

40 Candidate Generation Auto Indexing Traverse Query: let $doc:=doc("sample.xml") return for $book in return $book/title Candidates: QGM representation 1. // 2. book 4. book1 14

41 Candidate Generation Auto Indexing Traverse Query: let $doc:=doc("sample.xml") return for $book in return $book/title Candidates: 1. // 2. book 4. book1 QGM representation Over 18 enumeration rules and optimization rules 14

42 Candidate Selection Cost-Benefit Calculation Rank candidates by benefit Observe space limitation (Greedy algorithm) Search space pruning (merge indexes) Containment Problem: Comparing PCR sets, but index types and semantics may differ Reconfigure Index Set Schedule materialization and deletion jobs 15

43 Outline XML Index Types Statistics for Cost-based Index Selection Index Configuration Management Evaluation Summary & Outlook

44 Evaluation Workload Documents XMark documents (12MB and 112MB) TPoX collection (~250MB) Standard XML documents (dblp, treebank, psd, ) Benchmark Scenarios No Indexes Manual (element + content index) Manual + Self-tuning Self-tuning 17

45 Evaluation 18

46 Evaluation 18

47 Evaluation (2) 18

48 Evaluation (2) 18

49 Evaluation (2) (2 + 23) 18

50 Evaluation (2) (2 + 23) 18

51 Evaluation (2) (2 + 23) (23) 18

52 Evaluation Impact of parallel indexing Impact of aggressive indexing 19

53 Evaluation Overhead / Pruning effects Workload shifts % of workload Overhead in % 20

54 Summary Self-tuning of XML indexes causes new challenges to the ISP Path Synopsis use for storage, indexing, and managing Overhead of statistics and management (mostly) pays off Self-tuning of tuning frequency and pruning are effective Outlook Integrate Update workload Analyze workload shift reaction (stability vs. effectiveness) Evaluate XML warehouse scenarios Integrate data placement decision Combine with buffer tuning 21

55 Finish < />