MonetDB/XQuery: High Performance, Purely Relational XQuery Processing

Transcription

1 ADT 7 Lecture 4 MonetDB/XQuery: High Performance, Purely Relational XQuery Processing xquery.org/ xquery.org/ Stefan Manegold Stefan.Manegold@cwi.nl

2 Schedule Last lecture (7.9.): RDBMS back-end support for XML/XQuery (/): Document Representation (XPath Accelerator, Pre/Post plane) XPath navigation (Staircase Join) ADT 7

3 Schedule Last lecture (7.9.): RDBMS back-end support for XML/XQuery (/): Document Representation (XPath Accelerator, Pre/Post plane) XPath navigation (Staircase Join) Today: XQuery to Relational Algebra Compiler: Item- & Sequence- Representation Efficient FLWoR Evaluation (Loop-Lifting) Optimization RDBMS back-end support for XML/XQuery (/): Updateable Document Representation ADT 7

4 4 Source Language: XQuery Core ADT 7

5 5 Target Language: Relational Algebra ADT 7

6 6 Sequence Representation sequence = table of items add pos column for maintaining order ignore polymorphism for the moment ADT 7

7 7 Sequence Representation Item sequences, sequence order Item (, x, <a/>, ) Problems: Pos Node Atom NULL NULL x pre(a) NULL 4 NULL Polymorphic columns Redundant storage Copy overhead (especially with strings) Pos Item X pre(a) 4 ADT 7

8 8 Item Representation Item sequences, sequence order Item (, x, <a/>, ) Pos Item X pre(a) 4 Pos Node Atom NULL NULL x pre(a) NULL 4 NULL Pos 4 Item Int Pos Item X pre(a) 4 x ADT 7

9 9 Iterations ADT 7

10 Iterations ADT 7

11 Iterations ADT 7

12 Iterations ADT 7

13 Loop Lifting ADT 7

14 4 Loop Lifting ADT 7

15 5 Loop Lifting ADT 7

16 6 Loop Lifting ADT 7

17 7 Deriving loop ADT 7

18 Nested Scopes 8 ADT 7

19 Nested Scopes 9 ADT 7

20 Loop lifting ADT 7

21 Full Example join calc project ADT 7

22 XQuery On SQL Hosts [VLDB4] XQuery Construct sequence construction if then else for loops calculations list functions, e.g. fn:first() element construction XPath steps Relational Mapping A union B select(cond=true,b) union select(a=false,c) cartesian product project(a,x=expr(y,..yn) select(a,pos=) updates in temporary tables staircase join ADT 7

23 XMark Query 8 let $auction := doc("auctions.xml") ret urn for $p in $auction/site/people/person let $a := for $t in $auction/site/ closed_auctions/closed_auction whe re $t/buyer/@person = $p/@id ret urn $t ret urn <item person="{$p/name/text()}"> {count($a)} </item> ADT 7

24 Peephole Optimization 4 [Grust, XIME P 5] ADT 7

25 Peephole Optimization 5 [Grust, XIME P 5] ADT 7

26 Peephole Optimization 6 [Grust, XIME P 5] ADT 7

27 Peephole Optimization 7 [Grust, XIME P 5] ADT 7

28 Peephole Optimization 8 [Grust, XIME P 5] ADT 7

29 Peephole Optimization 9 [Grust, XIME P 5] Plan simplification ADT 7

30 Peephole Optimization [Boncz et al., SIGMOD 6] Sort Avoidance generated by DENSE RANK pos ORDER BY pos PARTITION BY iter by tracking secondary ordering properties [po s it er], [Wang&Cherniack, VLDB ] iter pos item X Y Z A B C [iter,pos] ADT 7

31 Peephole Optimization [Boncz et al., SIGMOD 6] Sort Avoidance generated by DENSE RANK pos ORDER BY pos PARTITION BY iter by tracking secondary ordering properties [po s it er], [Wang&Cherniack, VLDB ] iter pos item X Y Z A B C iter pos item X Y Z A B C [iter,pos] ADT 7

32 Peephole Optimization [Boncz et al., SIGMOD 6] Sort Avoidance generated by DENSE RANK pos ORDER BY pos PARTITION BY iter by tracking secondary ordering properties [po s it er], [Wang&Cherniack, VLDB ] hash based (streaming) DENSE_RANK iter pos item X Z A Y B C iter pos item X Z A Y B C [pos iter] ADT 7

33 Peephole Optimization [Boncz et al., SIGMOD 6] Sort Avoidance generated by DENSE RANK pos ORDER BY pos PARTITION BY iter by tracking secondary ordering properties [po s it er], [Wang&Cherniack, VLDB ] hash based (streaming) DENSE_RANK ADT 7

34 4 Peephole Optimization [Boncz et al., SIGMOD 6] Sort Avoidance generated by DENSE RANK pos ORDER BY pos PARTITION BY iter by tracking secondary ordering properties [pos i ter], [Wang&Cherniack, VLDB ] hash based (streaming) DENSE_RANK Sort Reduction if [iter,item] order is required, and [iter] only is present use a refine sort rather than a full sort. pipelinable sort ADT 7

35 5 Peephole Optimization [Boncz et al., SIGMOD 6] Sort Avoidance generated by DENSE RANK pos ORDER BY pos PARTITION BY iter by tracking secondary ordering properties [it em i ter], [Wang&Cherniack, VLDB ] hash based (streaming) DENSE_RANK Sort Reduction if [iter,item] order is required, and [iter] only is present use a refine sort rather than a full sort. pipelinable sort Join Detection detect cartesian products as multi valued dependencies in the precense of a theta selection theta join ADT 7

36 6 Loop lifted XPath Steps Many algorithms have been proposed & studied for XPath evaluation: Dataguide based, Structural Join, Staircase Join, Holistic Twig Join IN: sequence of context nodes in (doc order) OUT: sequence of document nodes (unique, in doc order) ADT 7

37 Loop lifted XPath Steps 7 In XQuery, expressions generally occur inside FLWR blocks, i.e. inside a for loop fo r $x in do c()// emplo yee $x/ ances tor:: depar tment Choice: call XPath algorithm N times, accessing document and index structures N times. use a loop lifted algorithm: IN: for each iteration, a sequence of context nodes OUT: for each iteration, a sequence of document nodes (per iteration unique, in doc order) ADT 7

38 Staircase join 8 document List of context nodes ADT 7

39 9 Loop lifted staircase join document List of context nodes document Multiple lists of context nodes Active stack Adapt: pruning, partitioning and skipping rules to correctly deal with multiple context sets ADT 7

40 Loop lifted staircase join 4 Results on the XMark queries: ADT 7

41 4 Performance Evaluation Extensive performance Evaluation on XMark data sizes KB, MB, MB, MB,.GB, GB MonetDB/XQuery, Galax.6, X Hive 6., Berkeley DB XML., exist 8GB RAM Extensive XMark performance Literature Overview IPSI XQ v..b, Dynamic Interval Encoding, Kweelt, QuiP, FluX, TurboXPath, Timber, Qizx/Open (Version.4/p), Saxon (Version 8.), BEA/XQRL, VX Crude comparison (normalized by CPU SPECint) ADT 7

42 4 XMark Benchmark MB XML GB XML Galax X Hive Berkeley DB XML exist ADT 7

43 4 What is MonetDB? Main memory based DBMS backend/kernel Developed at CWI since 99 Query intensive applications Data mining Data warehousing / decision support Multi media information retrieval (text, images, audio, video, XML,...) XML databases GIS part of Data Distilleries' products CWI spin off company (>GB) databases at ABN Amro, Postbank, Ohra, Spaarbeleg, FBTO, Centerparcs, Vodafone Nowadays: part of SPSS ADT 7

44 44 MonetDB: Motivation (/) Relational DBMS dominate the scene Oracle, SQLserver, DB databases a solved problem? ADT 7

45 45 MonetDB: Motivation (/) Relational DBMS dominate the scene Oracle, SQLserver, DB databases a solved problem? No! Problems: performance new query intensive applications (data mining, et al) extensibility new applications (GIS,text,image,audio,video,XML) ADT 7

46 46 MonetDB: Motivation (/) are relational DBMS fit for the job? developed in end 97 s begin 98 s ADT 7

47 47 MonetDB: Motivation (/) are relational DBMS fit for the job? developed in end 97 s begin 98 s hardware has changed CPUs get faster but more vulnerable capacity and bandwidth follows Moore s law latency becomes a bottleneck (I/O and RAM) ADT 7

48 48 MonetDB: Motivation (/) are relational DBMS fit for the job? developed in end 97 s begin 98 s hardware has changed CPUs get faster but more vulnerable capacity and bandwidth follows Moore s law latency becomes a bottleneck (I/O and RAM) applications have changed RDBMS tuned for transaction processing not query intensive only business domain ADT 7

49 Transactions (OLTP) 49 ADT 7

50 OLAP, Data Mining 5 ADT 7

51 How is MonetDB Different 5 full vertical fragmentation: always! everything in binary ( column) tables saves you from table scan hell in OLAP and Data Mining the RISC approach to databases simple data model, simple query language don t need (to pay for) a buffer manager => manage virtual memory explicit transaction management => DIY approach to ACID CPU and memory cache optimized programming team experienced in main memory DBMS techniques use of scientific programming optimizations (loop unrolling) Cache conscious data structures and algorithms ADT 7

52 5 MonetDB: Shopping List A quantum leap in performance requires a quantum leap in technology (and risk) Better support for non administrative applications, using: Multi model database kernel support Extensible data types, operators, accelerators Database hot set is memory resident (but scale to TB) Use simple data structures Index management should be automatic Algebraic language as the computational model Query optimization = strategic + tactic + operational optimization Dynamic optimization, parallelism, JIT compile link run Cooperative (application) transaction management Do not replicate the operating system ADT 7

53 Storing Relations in MonetDB 5 ADT 7

54 Relational Mapping 54 ADT 7

55 Object Oriented Mapping 55 ADT 7

56 56 BAT Data Structure BAT: binary association table BUN: binary unit Hash tables, T trees, R trees,... BUN heap: consecutive memory block (array) memory mapped file ADT 7

57 BAT Storage Optimizations 57 Dense ascending sequence ADT 7

58 BAT Property Management 58 type (physical) type number enum enumerated type flag dense dense ascending range sorted ascending head sorting constant all equal values align unique sequence id key no duplicates on column set no duplicates in BAT hash accelerator flag Ttree accelerator flag mirrored head=tail value count cardinality ADT 7

59 59 XML/XQuery Updates ADT 7

60 6 XML/ XQuery Updates ADT 7

61 XML/XQuery Updates 6 ADT 7

62 XML/XQuery Updates 6 ADT 7

63 6 XML/ XQuery Updates Staircase Join ADT 7

64 64 XML Storage Revisited a b c d e f g h i j pre post pre size level post = pre + size level pre size level 5 null null null Allow holes rid size level nid 5 4 null null N N null null N N N4 N5 N6 N7 N8 N9 Define logical pages ADT 7

65 65 XML Storage Revisited a b c d e f g h i j pre post pre size level post = pre + size level pre size level 5 null null null Allow holes rid = pre.swizzle rid ( null null N N null null N6 N7 N8 N9 N N N4 N5 Define logical pages page map size level nid ) ADT 7

66 66 XML Storage Revisited Update friendly rid table is append only rid tuples may be unused rid = autoincrement column MonetDB: rid not stored but computed (virtual oid) allows positional lookup/join Opportunity currently not exploited by other RDBMS rid size level nid 5 4 null null N N null null N6 N7 N8 N9 N N N4 N5 Occurs widely in our XQuery translation. ADT 7

67 67 XML Storage Revisited Update friendly rid table is append only rid tuples may be unused rid = autoincrement column MonetDB: rid not stored but computed (virtual oid) allows positional lookup/join Opportunity currently not exploited by other RDBMS rid size level nid 5 4 null null N N null null N6 N7 N8 N9 N N N4 N5 Occurs widely in our XQuery translation. ADT 7

68 68 MonetDB/XQuery Our own XML DBMS with (almost..) full XQuery support. Built purely on an RDBMS, namely MonetDB Future: also middleware support (PP!!) in AmbientDB Pathfinder compiler & staircase join (see later): Technical University Munich (Torsten Grust, et al.) Technical University Twente (Maurice van Keulen, et. al.) MonetDB High Performance DBMS CWI Amsterdam (Peter Boncz, Stefan Manegold,...) Pathfinder Compiler Relational Algebra RDBMS Useful for: (MonetDB) Large XML databases! Querying XML annotations (multimedia, forensic NFI) XQuery ADT 7

69 Current Projects 69 Value indeces & runtime optimization Code freeze, release this week Algebraic Query Optimization Some released, most still in the development version Distributed XQuery PP XQuery SOAP group communication, XQuery RPC VLDB'7 [Zhang, Boncz] Benchmarking beyond XMark ExpDB'6 Workshop [Manegold] Support for XML Interval Annotations XIME P'6 Workshop [Alink et al.]... ADT 7

70 Conclusions 7 Relational approach can be scalable & fast MonetDB/XQuery compares favorably with all other available systems Techniques that made it work Property driven peephole optimization Order & other properties Loop lifted XPath steps Evaluate Sets of context nodes in a single pass Support for dense (autoincrement) keys Positional lookup Background Information & Literature xquery.org xquery.org ADT 7