XML Stream Processing

Transcription

1 XML Stream Processing Dan Suciu Joint work with faculty, visitors and students at UW

2 Introduction This is a research project at UW Partially supported by MS Two parts: A free toolkit of command lines: xsort, xagg,... Research on XML stream processing this talk

3 The Problem Given: Large number of Xpath expressions Incoming stream of XML documents Decide for each document which expressions it matches

4 XML Data Stream XPath expressions Decisions <datasets> <dataset>... </datasets> [history/text()= recent ]/title ///* ///* ///*/text()="galaxy" /field /field

5 The Application(s) Selective Dissemination of Information [Berkeley] XML content routing [MIT] SOAP Message routing in Application Servers Typical scale: 10,000 to 1,000,000 Xpath expressions XML stream: 1KB/s? 1MB/s?

6 The Approaches Basic techniques NFA plus optimizations: Xfilter/Yfilter, XTrie DFA: we are doing this here Beyond the obvious SIX views

7 Background on NFA and DFA //a/b/a/a/b 0 a * [other] 0 a [other] 1 b [other] 01 b 2 a [other] 02 b a 3 a [other] 013 a a 4 [other] 014 b a b $X 5 $X 025 NFA DFA

8 Background on NFA and DFA //a/*/*/*/b 0 a * a 0 [other] 1 * 01 a [other] 2 * a [other] a [other] 3 * a [other] a [other] b b b b b $X $X $X $X $X.... NFA DFA (without back edges)

9 Background on NFA and DFA Issue: need to linearize Xpath expressions 1 Xpath expression with filters /catalog/product[@category="tools"][sales/@price > 200]/quantity 4 linear Xpath expressions /catalog/product/$y $Y/@category ="tools" $Y/sales/@price $Y/quantity Extra processing OK in trivial cases. Complex cases require more work (future) For now: assume all Xpath expressions are linear

10 XPath Basic NFA Evaluation NFAs [history/text()= recent ]/title ///* [history/text()= recent ]/title ///*/text()="galaxy" ///* ///*/text()="galaxy" /field /field [history/text()= recent ]/title ///* [history/text()= recent ]/title ///*/text()="galaxy" ///* ///*/text()="galaxy" /field... /field [history/text()= recent ]/title ///* [history/text()= recent ]/title ///*/text()="galaxy" ///* ///*/text()="galaxy" /field /text() /field /text() Current state ,66,102,4534,... <datasets> <dataset>... </datasets> SAX events 2,3,543,43,254 1,55,99,... STACK

11 XPath Basic DFA Evaluation DFAs [history/text()= recent ]/title ///* [history/text()= recent ]/title ///*/text()="galaxy" ///* ///*/text()="galaxy" /field /field [history/text()= recent ]/title ///* [history/text()= recent ]/title ///*/text()="galaxy" ///* ///*/text()="galaxy" /field... /field [history/text()= recent ]/title ///* [history/text()= recent ]/title ///*/text()="galaxy" ///* ///*/text()="galaxy" /field /text() /field /text() Current state 399 <datasets> <dataset>... </datasets> SAX events STACK

12 Comparison: Throughput in MB/s Throughput for 1k, 10k, 100k, 1000k XPEs [ prob(*)=10%, prob(//)=10% ] parser lazydfa(1k) lazydfa(10k) lazydfa(100k) lazydfa(1000t) xfilter(1k) xfilter(10k) xfilter(100k) xfilter(1000t) MB 10MB 15MB 20MB 25MB Total input size

13 Number of States in DFA Compute the DFA for 1,000,000 Xpath expressions???!!? 1 linear Xpath small DFA 1,000,000 linear Xpaths HUGE DFA

14 Number of States in DFA //section//footnote //figure//footnote //table//footnote //abstract//footnote n Xpath expressions 2 n states Solution: lazy DFA!

15 Number of States in the lazy DFA Real XML data Synthetic XML data Non-recursive or data-style recursive DTDs Document-style recursive DTD Theorem DFA is small Theorem DFA is small Theorem DFA is small DFA is HUGE

16 Number of DFA States - SYNTHETIC Data k XPEs 10k XPEs 100k XPEs simple prov ebbpss protein nasa treebank

17 Number of DFA States - REAL Data k XPEs 10k XPEs 100k XPEs protein nasa treebank

18 Beyond the Obvious I: Stream IndeX (SIX) Main observation: Parsing is major bottleneck Skip portions of the XML document avoid parsing and processing

19 Stream IndeX (SIX) XML SIX <bib> <bib> <book> <book> <publisher> <publisher> Addison-Wesley Addison-Wesley </publisher> </publisher> <author> <author> Serge Serge Abiteboul Abiteboul </author> </author> <author> <author> <first-name> <first-name> Rick Rick </first-name> </first-name> <last-name> <last-name> Hull Hull </last-name> </last-name> </author> </author> <author> <author> Victor Victor Vianu Vianu </author> </author> <title> <title> Foundations Foundations of of Databases Databases </title> </title> <year> <year> </year> </year> </book> </book> <book <book price= 55 > price= 55 > <publisher> <publisher> Freeman Freeman </publisher> </publisher> <author> <author> Jeffrey Jeffrey D. D. Ullman Ullman </author> </author> <title> <title> Principles Principles of of Database Database and and Knowledge Knowledge Base Base Systems Systems </title> </title> <year> <year> </year> </year> </book> </book> </bib> </bib>... bib book publisher author author beginoffset endoffset

20 Stream IndeX (SIX) API for SIX: skip(k), where k >= 0 skips to the end of the k th surrounding element Uses beginoffset to sync with the XML doc Uses endoffset to skip

21 Stream IndeX (SIX) SIX SIX SIX <datasets> <dataset>... </datasets> <datasets> <dataset>... </datasets> <datasets> <dataset>... </datasets> XML XML XML The SIX stream is about 6% of the data stream And can be made MUCH smaller

22 Throughput improvements from SIX (stable) MB/s Theta=3% (SIX) Theta=3% Theta=8% (SIX) Theta=8% Theta=14% (SIX) Theta=14% XML stream (MB)

23 Beyond the obvious II: View Selections On-going work: View selections header header header header <datasets> <dataset>... </datasets> <datasets> <dataset>... </datasets> <datasets> <dataset>... </datasets> XML XML XML 100x speedup On a hit

24 Conclusions Two ideas: Computing the DFA is possible! Use extra info to further speedup: SIX, Headers Issues: Extend DFAs to filters: process events How to represent SIX or Headers in XML

25 Msdn.microsoft.com/webservices contact