XML Data Streams. XML Stream Processing. XML Stream Processing. Yanlei Diao. University of Massachusetts Amherst

Transcription

1 XML Stream Proessing Yanlei Diao University of Massahusetts Amherst XML Data Streams XML is the wire format for data exhanged online. Purhase orders News feeds Stok tikers Transations of online autions Network monitoring data XML Stream Proessing XML data ontinuously arrives from external soures, queries are evaluated every time a new data item is reeived. A key feature of stream proessing is the ability to proess as it arrives. Natural fit for XML message brokering and web servies where messages need to be filtered and transformed on-the-fly. XML stream proessing allows query exeution to start before a message is ompletely reeived. Short delay in produing results. No need to buffer the entire message for query proessing. Both are ruial when input messages are large, e.g. the equivalent of a database s worth of data.

2 Event-Based Parsing XML stream proessing is often performed on the granularity of an XML parsing event produed by an event-based API. <?xml version=".0"?> <report> < id= intro diffiulty= easy > <>Pub-Sub</> < diffiulty= easy > < soure= g.jpg > <>XML Proessing</> </> </> < soure= g.jpg > <>Salability</> </> </> </report> < Start Doument < Start Element: report < Start Element: < Start Element: Charaters: Pub/Sub > End Element: < Start Element: < Start Element: < Start Element: Charaters: XML Proessing > End Element: > End Element: > End Element: > End Element: > End Element: report Yanlei Diao, University of Massahusetts Amherst > End //0 Doument Mathing a Single Path Expression XFilter [Altinel&Franklin 00], YFilter [Diao et al. 0], Tukwila [Ives et al. 0] Simple paths: ( ( / // ) (ElementName * ) )+ A simple path an be transformed to a regular expression Let Σ = set of element names: / is translated to the onatenation operator // is translated to Σ* * is translated to Σ /a//b an be translated to a Σ* b A finite automaton (FA) for eah path: mapping loation steps to mahine states. Query Compilation Map loation steps to automaton fragments a Loation /a Automaton steps /* fragments * //a * a For multi-path proessing; o.w. not needed Conatenate automaton fragments for loation steps in a query Query /a//b a a * b * b Is this Automaton deterministi or non-deterministi?

3 Event-Driven Query Exeution * Query exeution retrieves all mathes of a path expression. Event-driven exeution of FA: Parsing events (esp. start of elements) drive the FA exeution. Elements that trigger transitions to the aepting states are returned. < Start Doument < Start Element: report < Start Element: < Start Element: Charaters: Pub/Sub > End Element: < Start Element: < Start Element: < Start Element: Charaters: XML Proessing > End Element: > End Element: > End Element: > End Element: > End Element: report > End Doument Run FA over XML with nested struture, retrieve all results Approah : shred XML into a set of linear paths Approah : augment FA exeution with baktraking Multi-Query Proessing Problem: evaluate a set Q = Q,, Qn of path queries against eah inoming XML doument. Brute fore: iterate the query set, one query at a time Indexing of queries: inverse problem of traditional query proessing Traditional DB: Data is stored persistently; queries exeuted against the data. Indexes of data enable seletive searhes of the data. XML stream proessing: Queries are persistently stored; douments or their parsing events drive the mathing of queries. Indexes of queries enable seletive mathing of douments to queries. Sharing of proessing: ommonalities exist among queries. Shared proessing avoids redundant work. Construting the Combined FSM YFilter [Diao et al. 0] builds a ombined FA for all paths. Complete prefix sharing among paths. Moore mahine with output: aepting states partition of query ids. Nondeterministi Finite Automaton (NFA)-based implementation: a small mahine size, flexible, easy to maintain, et. Q=/a/b Q=/a/ Q=/a/b/ Q=/a//b/ Q=/a/*/b Q=/a// Q=/a/*/*/ Q=/a/b/ a b * {Q} * b {Q} {Q, {Q} Q} {Q} {Q} {Q} {Q} *

4 Exeution Algorithm YFilter uses a stak mehanism to handle XML. Baktraking in the NFA. No repeated work for the same element. {Q} {Q, Q} NFA b {Q} {Q} a * b {Q} * 9 {Q} 0 {Q} * Runtime Stak 9 initial read <a> read <b> read <> math Q math Q Q Q Q Q 9 read </> An XML fragment <a> <b> <> </> 0 9 Implementation Choies Non-Deterministi Automata (NFA) versus Deterministi Automata (DFA) NFA: small mahine, large numbers of transitions per element DFA: potentially large mahine, one transition per element Worse-ase omparison for regular expressions: NFA DFA Single regular expression of length n Proessing omplexity O(n ) O() O(n) Mahine size O(n) O( n ) O(n m) O(nm) O(nm) O() m regular expressions ompiled together Proessing omplexity Mahine size O( nm ) Restrited path expressions: YFilter speifi Possible in pratie? Eager DFA Green et al. studied the size of DFA for the restrited set of path expressions [Green et al. 0] Eager DFA: Query ompile time, translates from NFA to DFA Creates a state for every possible situation that runtime may require Single path: Linear for //e/e/e/e/e Exponential for //e/*/*/*/e Multiple paths: Exponential for //e//b, //e//b,, //e//b

5 Example of an Eager DFA //a/*/*//d a a b b d a b a b d Need to remember all the A s in three onseutive haraters, as different ombinations of As may yield different results. A Not A Not A or C A Not A Not A A Start A Not A Not A A Not A or D C C C C D Aept The DFA size is O( w+ ) where w is the number of *. Lazy DFA Lazy DFA is onstruted at run time, on demand Initially, it has a single state Whenever it attempts to make a transition into a missing state, it omputes it and updates the transition Hope: only a small set of the DFA states is needed. Exploits DTD to derive upper bounds Lazy DFA (ontd.) A DTD graph is simple, if the only loops are self-loops. Theorem: the size of a lazy DFA is exponential only in the maximal number of simple yles a path an interset. Not exponential in # of paths! DTD graph More omplex reursive DTDs: e.g. table ontains list and list ontains table Even a lazy DFA grows large

6 Prediates in Path Expressions Prediates an address attributes, text data, or positions of elements. Value of an attribute in an element, e.g., = easy ]. Text data of an element, e.g., ///[text()= XPath ]. Position of an element, e.g., ///[text()= XPath ][]. Prediate Evaluation Extend the NFA inluding additional states representing suessful evaluation of prediates and transitions to them Potential problems A potentially huge inrease in the mahine size Destroy sharing of path expressions Reent work [Gupta & Suiu 00]: Possible to build an effiient pushdown automaton using lazy onstrution if No mixed ontent, suh as <a> <b> </b> </a>. Can afford to periodially rebuild the automaton from srath. Can afford to train the automaton in eah onstrution. YFilter: Mapping XML to Relational XQuery is muh more omplex than regular expressions. Leverage effiient relational proessing XQuery stream proessing = relational operations on pathtuple streams! report P: //// P: //// {P} * path-tuple streams * Shared Path Mathing Engine parsing events {P}

7 Example Query Plan for FWR Q: for F: // $s in easy ] W:/// where $s/ = Pub/Sub W://// and $s// = XML proessing return <> { $s//// } R:////// R://// { $s// } </> Query i ΠDupElim < < σ σ σ OuterJoin-Selet ΠDupElim ΠDupElim Push all paths into the path engine. F W W R R An external (post-proessing) plan for eah query: Shared Path Mathing Engine Seletion: evaluates value-based prediates. Projetion: projets onto speifi fields and removes dupliates. Semijoin: handles orrelations between for and where paths, finds query mathes. Outerjoin-Selet: handles orrelations between for and return paths, generates query results. Buffering in XML Stream Proessing Buffering in XQuery stream proessing Whether an element belongs to the result set is unertain as it depends on prediates that have not been evaluated Goal: avoid materialization and minimize buffering Issues to address: What queries require buffering? What elements to buffer? When and how to prune dynami data strutures? Buffered elements State in any dynami data strutures Buffering in XML Stream Proessing //[= XML ] Requires buffering? Yes What element to buffer? When to prune? until the end of is enountered, or when no more an our in this. (Use easy g.jpg g.jpg Salability XML proessing

8 Buffering in XML Stream Proessing //[= XML ]/ Requires buffering? Yes What element to buffer? When to prune? until a mathes the prediate, or when no more an our in this (use DTD!), or the end of is easy g.jpg g.jpg Salability XML proessing Buffering in XML Stream Proessing //[ontains(.//, XML )]// Requires buffering? Yes What element to buffer??,? When to prune? until the prediate is true, or when no more an our in this (use DTD!), or the end of is easy g.jpg report XML g.jpg Salability