Traditional Query Processing. Query Processing. Meta-Data for Optimization. Query Optimization Steps. Algebraic Transformation Predicate Pushdown

Transcription

1 Traditional Query Processing 1. Query optimization buyer Query Processing Be Adaptive SELECT S.s FROM Purchase P, Person Q WHERE P.buyer=Q. AND Q.city= seattle AND Q.phone > Query execution σ City= seattle phone> Buyer= (Simple Nested Loops) Purchase Person (Table scan) (Index scan) Meta-Data for Optimization Sizes of tables in the database Selectivity information: How many rows per value in a column How many rows per range Cardinality of certain views Indexes on the data: B-trees, hash indices, sorting order Query Optimization Steps Algebraic transformations Can be very complex Select ordering of joins Many possible orders. Can t explore them all Select algorithm for each node in the query plan Many possible implementations of relational operators Algebraic Transformation Predicate Pushdown Determining Join Ordering s bid=100 rating > 5 s R 1 R 2. R n Join tree: sid=sid Reserves sid=sid Sailors (Scan; write to temp T1) bid=100 Reserves (Scan; rating > 5 write to temp T2) Sailors The earlier we process selections, less tuples we need to manipulate higher up in the tree. Disadvantages? R 3 R 1 R 2 R 4 A join tree represents a plan. 1

2 Left Deep Trees Good for pipelining Hash-Join Partition both relations using hash fn h: R tuples in partition i will only match S tuples in partition i. Original Relation... OUTPUT INPUT 2 hash function h B-1 1 Partitions 1 2 B-1 Disk B main memory buffers Disk R 3 R 1 R 5 R 2 R 4 Read in a partition of R, hash it using h2 (<> h!). Scan matching partition of S, search for matches. Partitions of R & S Disk hash fn h2 Hash table for partition Ri (k < B-1 pages) h2 Input buffer for Si Output buffer B main memory buffers Join Result Disk Cost of Hash-Join In partitioning phase, read+write both relations; 2(M+N). In matching phase, read both relations; M+N I/Os. Sort-Merge Join Sort R and S on the join column, then scan them to do a ``merge on the join column. Advance scan of R until current R-tuple >= current S tuple, then advance scan of S until current S- tuple >= current R tuple; do this until current R tuple = current S tuple. At this point, all R tuples with same value and all S tuples with same value match; output <r, s> for all pairs of such tuples. Then resume scanning R and S. Cost of Sort-Merge Join But in Data Integration R is scanned once; each S is scanned once per matching R tuple. Cost: M log M + N log N + (M+N) The cost of scanning, M+N, could be M*N (unlikely!) Sort-Merge Join vs. Hash Join: Given a minimum amount of memory both have a cost of 3(M+N) I/Os. Hash Join superior on this count if relation sizes differ greatly. Also, Hash Join shown to be highly parallelizable. Few statistics, if any autonomous sources Unanticipated delays and failures network-bound sources A query plan can look good but not work well Desiderata: optimize time to first tuples Reduce load on sources 2

3 Adaptive Query Processing Double Pipelined Join Reconsider query plan during execution! But when? How? Operators better suited to dynamic environments Double pipelined join. Hash Join Partially pipelined: no output until inner read Asymmetric (inner vs. outer) optimization requires source behavior knowledge Double Pipelined Hash Join Outputs data immediately Symmetric requires less source knowledge to optimize Re-optimization (1) Re-optimization (2) Re-optimize at materialization points R 4 Insert conditionals into plan: R 4 R 2 R 2 R 5 R 5 R 3 R 1 R 3 R 1 if C then join with R 5 otherwise, R 2 Other Radical Options Eddies: a separate plan for every tuple Depends on speed and selectivity But decisions are very local Parallel execution: Try out 2(+) plans and see which goes better Terminate the other one. Multiple Challenges Don t increase the cost of planning Minimize wasted work Tradeoff local vs. global decisions Don t get stuck because of one bad decision Partition plan vs. data? 3

4 R 0 S 0 T 0 R 0 S 0 R 0 Adaptive Data Partitioning S 0 R S T R 1 S 1 T 1 T 0 S 1 T R 1 1 R 0 S 0 S 1 T 1 Exclude R 0 S 0 R 0 S 0 R 1 Exclude R 0 S 0 T 0, R 1 S 1 T 1 T 0 T 1 S 1 Query Processing Summary We don t have the usual statistics Hence, need to be adaptive Adaptivity needs to trade off several factors From a theoretical viewpoint, this is still an open problem. And what if results are assumed to be approximate? R S T XML and Data Integration XML The Data Integration Appetizer XML: whetted the integration appetites We have the syntax Now just solve the silly semantics problems Don t bother: we ll all standardize on DTDs. XML had a significant role on the data integration industry and research. XML <bibliography> <book> <title> Foundations </title> </book> </bibliography> <author> Abiteboul </author> <author> Hull </author> <author> Vianu </author> <publisher> Addison Wesley </publisher> <year> 1995 </year> XML describes the content vs. presentation XML Terminology tags: book, title, author, start tag: <book>, end tag: </book> elements: <book> <book>,<author> </author> elements are nested empty element: <red></red> abbrv. <red/> an XML document: single root element well formed XML document: if it has matching tags 4

5 More XML: Attributes <book price = 55 currency = USD > <title> Foundations of Databases </title> <author> Abiteboul </author> <year> 1995 </year> </book> attributes are alternative ways to represent data More XML: Oids and References <person id= o555 > <> Jane </> <person id= o456 > <> Mary </> <children idref= o123 o555 /> <person id= o123 mother= o456 ><>John</> oids and references in XML are just syntax XML Semantics: a Tree! <data> <person id= o555 > <> Mary </> <address> <street> Maple </street> <no> 345 </no> <city> Seattle </city> </address> <person> <> John </> <address> Thailand </address> <phone> </phone> </data> id o555 Attribute node Mary person address street no city data Maple 345 Seattle Order matters!!! John person address Thai Text node Element node phone XML Data XML is self-describing Schema elements become part of the data Relational schema: persons(,phone) In XML <persons>, <>, <phone> are part of the data, and are repeated many times Consequence: XML is much more flexible XML = semistructured data Relational Data as XML person XML: person n a m e p h o n e J o h n S u e D i c k row row row phone phone phone John 3634 Sue 6343 Dick 6363 <person> <row> <>John</> <phone> 3634</phone></row> <row> <>Sue</> <phone> 6343</phone> <row> <>Dick</> <phone> 6363</phone></row> XML is Semi-structured Data Missing attributes: Could represent in a table with nulls <person> <> John</> <phone>1234</phone> <person> <>Joe</> no phone! John Joe phone

6 XML is Semi-structured Data XML is Semi-structured Data Repeated attributes <person> <> Mary</> <phone>2345</phone> <phone>3456</phone> two phones! Attributes with different types in different objects <person> <> <first> John </first> <last> Smith </last> </> <phone>1234</phone> structured! Impossible in tables: Mary phone ??? Nested collections (no 1NF) Heterogeneous collections: <db> contains both <book>s and <publisher>s Document Type Definitions DTD part of the original XML specification an XML document may have a DTD XML document: well-formed = if tags are correctly closed Valid = if it has a DTD and conforms to it validation is useful in data exchange Very Simple DTD <!DOCTYPE company [ <!ELEMENT company ((person product)*)> <!ELEMENT person (ssn,, office, phone?)> <!ELEMENT ssn (#PCDATA)> <!ELEMENT (#PCDATA)> <!ELEMENT office (#PCDATA)> <!ELEMENT phone (#PCDATA)> <!ELEMENT product (pid,, description?)> <!ELEMENT pid (#PCDATA)> <!ELEMENT description (#PCDATA)> ]> Very Simple DTD Example of valid XML document: <company> <person> <ssn> </ssn> <> John </> <office> B432 </office> <phone> 1234 </phone> <person> <ssn> </ssn> <> Jim </> <office> B123 </office> <product>... </product>... </company> Querying XML Data XPath = simple navigation through the tree = the SQL of XML 6

7 Sample Data for Queries <bib> <book> <publisher> Addison-Wesley </publisher> <author> Serge Abiteboul </author> <author> <first-> Rick </first-> <last-> Hull </last-> </author> <author> Victor Vianu </author> <title> Foundations of Databases </title> <year> 1995 </year> </book> <book price= 55 > <publisher> Freeman </publisher> <author> Jeffrey D. Ullman </author> <title> Principles of Database and Knowledge Base Systems </title> <year> 1998 </year> </book> </bib> Data Model for XPath The root bib The root element book book publisher author.... Addison-Wesley Serge Abiteboul XPath: Simple Expressions /bib/book/year Result: <year> 1995 </year> <year> 1998 </year> /bib/paper/year XPath: Restricted Kleene Closure //author Result:<author> Serge Abiteboul </author> <author> <first-> Rick </first-> <last-> Hull </last-> </author> <author> Victor Vianu </author> <author> Jeffrey D. Ullman </author> Result: empty (there were no papers) /bib//first- Result: <first-> Rick </first-> Xpath: Text Nodes /bib/book/author/text() Result: Serge Abiteboul Jeffrey D. Ullman Rick Hull doesn t appear because he has first, last Functions in XPath: text() = matches the text value node() = matches any node (= * or text()) () = returns the of the current tag //author/* Xpath: Wildcard Result: <first-> Rick </first-> <last-> Hull </last-> * Matches any element 7

8 Result: 55 Xpath: Attribute means that price is has to be an attribute XPath: Predicates /bib/book/author[first] Result: <author> <first-> Rick </first-> <last-> Hull </last-> </author> XPath: More Predicates Xpath: More Predicates /bib/book/author[first][address[//zip][city]]/last < 60 ] Result: <last> </last> <last> </last> /bib/book[author/@age < 25 ] /bib/book[author/text()] Xpath: Summary bib matches a bib element * matches any element / matches the root element /bib bib/paper bib//paper //paper paper bib/book/@price matches a bib element under root matches a paper in bib matches a paper in bib, at any depth matches a paper at any depth matches a paper or a book matches a price attribute matches price attribute in book, in bib bib/book/[@price< 55 ]/author/last matches XML: Query Language History XPath was there Nobody thought to join XML data DB community (circa 95-96) Semi-structured data: Lorel, UnQL, StruQL XML-QL: applied StruQL to XML XML paper-mill starts working formed (1998) But not in place when some of us needed it. 8

9 Based on Quilt, which is based on XML- QL Uses XPath to express more complex queries FLWR ( Flower ) Expressions FOR... LET... WHERE... RETURN... Find all book titles published after 1995: FOR $x IN document("bib.xml")/bib/book WHERE $x/year > 1995 RETURN { $x/title } Find book titles by the coauthors of Database Theory : FOR $x IN bib/book[title/text() = Database Theory ]/author $y IN bib/book[author/text() = $x/text()]/title RETURN <answer> { $y/text() } </answer> Result: <title> abc </title> <title> def </title> <title> ghi </title> The answer will contain duplicates! Result: <answer> abc </ answer > < answer > def </ answer > < answer > ghi </ answer > Same as before, but eliminate duplicates: FOR $x IN bib/book[title/text() = Database Theory ]/author $y IN distinct(bib/book[author/text() = $x/text()]/title) RETURN <answer> { $y/text() } </answer> distinct = a function that eliminates duplicates Result: <answer> abc </ answer > < answer > def </ answer > < answer > ghi </ answer > : Nesting For each author of a book by Morgan Kaufmann, list all books she published: FOR $a IN distinct(document("bib.xml") /bib/book[publisher= Morgan Kaufmann ]/author) RETURN <result> { $a, FOR $t IN /bib/book[author=$a]/title RETURN $t } </result> 9

10 Result: <result> <author>jones</author> <title> abc </title> <title> def </title> </result> <result> <author> Smith </author> <title> ghi </title> </result> FOR $x in expr -- binds $x to each value in the list expr LET $x = expr -- binds $x to the entire list expr Useful for common subexpressions and for aggregations <big_publishers> FOR $p IN distinct(document("bib.xml")//publisher) LET $b := document("bib.xml")/book[publisher = $p] WHERE count($b) > 100 RETURN { $p } </big_publishers> Find books whose price is larger than average: LET $a=avg( /bib/book/price) FOR $b in /bib/book WHERE $b/price > $a RETURN { $b } count = a (aggregate) function that returns the number of elms Let s try to write this in SQL Summary: FOR-LET-WHERE-RETURN = FLWR FOR/LET Clauses WHERE Clause RETURN Clause List of tuples List of tuples XML & Data Integration Need to extend every aspect of the system to handle XML Great if you re looking for papers to write! Schema mapping languages AQUV, containment Query processing Storage Update language Instance of Xquery data model 10

11 An XML Mapping Language <students> { for $p in /projects/project, $pmember in $p/, $studname in $pmember//text(), $pos in $pmember/position/text() where $pos = student return <student> <> { $studname } </> {for $sp in /projects/project, $pname in $sp//text(), $mname in $sp///text() where $mname = $studname return <project> { $pname } </project> } </student> } </students> Target schema (UW): Source schema (UPenn) students student advisor projects project project position What is Query Containment? Input D Output Q(D) Instance containment Query containment Relational Query Sets of tuples A set of tuples Q(D) Q (D) -- Subset Q Q -- for every input D, Q(D) Q (D) What is Query Containment? Example An XML Instance Input D Output Q(D) Instance containment Query containment Relational Query Sets of tuples A set of tuples Q(D) Q (D) -- Subset Q Q -- for every input D, Q(D) Q (D) XML Query An XML instance tree An XML instance tree Q(D) Q (D) -- Tree homomorphism Q Q --for every input D, Q(D) Q (D) D: <project> <></> </project> <project> <></> </project> project project : leaf nodes Example An XML Query Example Another XML Query Q1: FOR $x IN /project RETURN <>{ FOR $y IN $x/ RETURN <>{ FOR $z IN $y[text()= ] RETURN </> FOR $z IN $y[text()= ] RETURN </> }</> }</> D: project Q1(D): project Q2: FOR $x IN /project RETURN <>{ FOR $y IN /project/ RETURN <>{ FOR $z IN $y[text()= ] RETURN </> FOR $z IN $y[text()= ] RETURN </> }</> }</> D: Q2(D): project project 11

12 Example -- Tree Homomorphism and XML Instance Containment Q1(D): Q1(D): Q2(D): Q1(D) Q2(D) Q2(D): Q2(D) X Q1(D) Complexity for XPaths Restriction With //, * With *, [] (branch) With //, [] With //, *, [] With //, [], Equi-join on tags With //, *, [], Equi-join on tags, disjunction Regular Expression Complexity PTIME PTIME PTIME CO-NPC NPC 2 P -complete PSPACE-hard Citation [Amer-Yahia et al., 2001] [Milo and Suciu, 1999] [Miklao and Suciu, 2002] [Deutsch and Tannen, 2001] [Florescu et al., 1998] Complexity for Nested XPaths Restriction With //, * With *, [] (branch) With //, [] With //, *, [] With //, [], Equi-join on tags With //, *, [], Equi-join on tags, disjunction Regular Expression XPATH-Complexity PTIME PTIME PTIME CO-NPC NPC 2 P -complete PSPACE-hard Nested-Query Complexity (with fixed depth) CO-NPC CO-NPC 2 P -complete 2 P -complete PSPACE-hard? (not sure) Q1: <result> faculty Containment Algorithm(1): Query Head Embedding db people position $pos= professor <people> Query Head student <student> <> {$student} advisee $advisee = $student <advisor> {$} Query head embedding: <result> <people> db <faculty> <> people faculty student <student> {$} {$student} faculty advisee $advisee = $student <advisor> {$} :Q2 <result> Containment Algorithm(2): Query Body Embedding db db faculty people people <people> Q1: position $pos= professor Query body embedding: :Q2 <people> <faculty> student <student> <> {$student} advisee $advisee = $student <advisor> {$} <result> student <student> <> {$student} faculty advisee $advisee = $student <advisor> {$} XML Summary Great for data integration: People could envision sharing data Great for research: XML most popular topic at conferences for a few years Great for industry: Many services built on XML Let s declare victory and move on.. 12

13 Model Management [Bernstein et al.] Generic infrastructure for managing schemas and mappings: Manipulate models and mappings as bulk objects Operators to create & compose mappings, merge & diff models Short operator scripts can solve schema integration, schema evolution, reverse engineering, etc. First challenge: semantics of operators. 13