CSCC43 Introduction to Databases. XML Query Languages CSCC43

Transcription

1 Introduction to Databases XML Query Languages 1

2 Outline Semistructured data Introduction to XML Introduction to DTDs XPath core query language for XML XQuery full-featured query language for XML 2

3 Semistructured Data An HTML document to be displayed on the Web: <dt>name: John Doe <dd>id: s <dd>address: <ul> <li>number: 123</li> <li>street: Main</li> </ul> </dt> <dt>name: Joe Public <dd>id: s </dt> HTML does not distinguish between attributes and values 3

4 Why study XML? Huge demands for data exchange across platforms across enterprises Huge demands for data integration heterogeneous data sources data sources distributed across different locations XML (extensible Markup Language) has become the prime standard for data exchange on the Web and a uniform data model for integrated data. RDB OODB 4

5 Why not HTML? An Example Amazon publishes a catalog for books on sale Data source: a relational database Publishing: HTML pages generated from the relational database Customers want to query the catalog data: They can only access the published Web pages (and hence need a parser) They are only interested in information about books on Databases -- in SQL: select B from book B where B.title LIKE Database% 5

6 What is wrong with HTML? <h3> Books </h3> Databases <ol> <li> <b>database Design for Mere Mortals </b>michael J. Hernandez<br> <i>mar </i> <li> <b>beginning Database Design: From Novice to Professional </b> Clare Churcher <br> </ol> 6

7 What is wrong with HTML? A minor format change to the HTML document may break the parser and yield wrong answer to the query Why? HTML tags are predefined and fixed describing display format rather than structure of data HTML is good for presentation (human friendly), but does not help automatic data extraction by means of programs (queries) 7

8 An XML solution <books> <book > <title>database Design for Mere Mortals </title> <author>michael J. Hernandez</author> <date>13/03/2003 </date> </book> <book id = B2 > <title>beginning Database Design: From Novice to Professional</title> <author>clare Churcher</author> </book> </books> 8

9 Semistructured Data cont d To make the previous student list suitable for machine consumption on the Web, it should have these characteristics: Be object-like Be schemaless (not guaranteed to conform exactly to any schema, but different objects have some commonality among themselves) Be self-describing (some schema-like information, like attribute names, is part of data itself) Data with these characteristics are referred to as semistructured. 9

10 What is Self-describing Data? Non-self-describing (relational, object-oriented): Data part: (#123, [ Students, {[ John, s , [123, Main St ]], [ Joe, s , [321, Pine St ]] } ] ) Schema part: PersonList[ ListName: String, Contents: [ Name: String, Id: String, Address: [Number: Integer, Street: String] ] ] 10

11 What is Self-Describing Data? cont d Self-describing: Attribute names embedded in the data itself, but are distinguished from values Doesn t need schema to figure out what is what (but schema might be useful nonetheless) (#12345, [ListName: Students, Contents: { [ Name: John Doe, Id: s , Address: [Number: 123, Street: Main St. ] ], [Name: Joe Public, Id: s , Address: [Number: 321, Street: Pine St. ] ] } ] ) 11

12 Overview of XML Like HTML, but any number of different tags can be used (up to the document author) extensible Unlike HTML, no semantics behind the tags For instance, HTML s <table> </table> means: render contents as a table; in XML: doesn t mean anything special Some semantics can be specified using XML Schema (types); some using stylesheets (browser rendering) Unlike HTML, is intolerant to bugs Browsers will render buggy HTML pages XML processors are not supposed to process buggy XML documents 12

13 Root Root element elements Example <?xml version= 1.0?> attributes <PersonList Type= Student Date= > <Title Value= Student List /> <Person> </Person> <Person> </Person> </PersonList> Empty element Element (or tag) names Elements are nested Root element contains all others 13

14 More Terminology Opening tag <Person Name = John Id = s > Content of Person John is a nice fellow <Address> <Number>21</Number> <Street>Main St.</Street> </Address> </Person> standalone text, not very useful as data, non-uniform Child of Address, Descendant of Person Nested element, child of Person Parent of Address, Ancestor of number Closing tag: What is open must be closed 14

15 Well-formed XML Documents Must have a root element Every opening tag must have matching closing tag Elements must be properly nested <foo><bar></foo></bar> is a no-no An attribute name can occur at most once in an opening tag. If it occurs, It must have an explicitly specified value (Boolean attrs, like in HTML, are not allowed) The value must be quoted (with or ) XML processors are not supposed to try and fix illformed documents (unlike HTML browsers) 15

16 Tree Structure of XML <bookstore> <book category="cooking"> <title lang="en">everyday Italian</title> <author>giada De Laurentiis</author> <year>2005</year> <price>30.00</price> </book> <book category="children"> <title lang="en">harry Potter</title> <author>j K. Rowling</author> <year>2005</year> <price>29.99</price> </book> <book category="web"> <title lang="en">learning XML</title> <author>erik T. Ray</author> <year>2003</year> <price>39.95</price> </book> </bookstore> 16

17 Identifying and Referencing with Attributes An attribute can be declared to have type: ID unique identifier of an element If attr1 & attr2 are both of type ID, then it is illegal to have <something attr1= abc > <somethingelse attr2= abc > within the same document IDREF references a unique element with matching ID attribute If attr1 has type ID and attr2 has type IDREF then we can have: <something attr1= abc > <somethingelse attr2= abc > IDREFS a list of references, if attr1 is ID and attr2 is IDREFS, then we can have <something attr1= abc > <somethingelse attr1= cde >...<someotherthing attr2= abc cde > 17

18 Document Type Definition (DTD) A DTD is a grammar specification for an XML document DTDs are optional don t need to be specified If specified, DTD can be part of the document (at the top), or it can be given as a URL A document that conforms (i.e., parses) w.r.t. its DTD is said to be valid XML processors are not required to check validity, even if DTD is specified But they are required to test well-formedness 18

19 DTDs (cont d) DTD specified as part of a document: <?xml version= 1.0?> <!DOCTYPE Book [ ]> <Book> </Book> DTD specified as a standalone thing <?xml version= 1.0?> <!DOCTYPE Book > <Book> </Book> 19

20 DTD Components <!ELEMENT elt-name ( contents )/EMPTY/ANY > <!ATTLIST elt-name attr-name CDATA/ID/IDREF/IDREFS #IMPLIED/#REQUIRED > Optional/mandatory Element s contents An attr for elt Type of attribute Can define other things, like macros (called entities in the XML jargon) 20

21 DTD Example <!DOCTYPE Report [ <!ELEMENT Report (Students, Classes, Courses)> <!ELEMENT Students (Student*)> <!ELEMENT Classes (Class*)> <!ELEMENT Courses (Course*)> <!ELEMENT Student (Name, Status, CrsTaken*)> <!ELEMENT Name (First,Last)> <!ELEMENT First (#PCDATA)> <!ELEMENT CrsTaken EMPTY> <!ELEMENT Class (CrsCode, Semester, ClassRoster)> <!ELEMENT Course (CrsName)> <!ATTLIST Report Date CDATA #IMPLIED> <!ATTLIST Student StudId ID #REQUIRED> <!ATTLIST Course CrsCode ID #REQUIRED> <!ATTLIST CrsTaken CrsCode IDREF #REQUIRED Semester CDATA #REQUIRED > <!ATTLIST ClassRoster Members IDREFS #IMPLIED> ]> Zero or more Has text content Empty element, no content Same attribute in different elements 21

22 Example: Report Document with Cross-References <?xml version= 1.0?> <Report Date= > <Students> <Student StudId= s > <Name><First>John</First><Last>Doe</Last></Name> <Status>U2</Status> <CrsTaken CrsCode= CS308 Semester= F1997 /> <CrsTaken CrsCode= MAT123 Semester= F1997 /> </Student> <Student StudId= s > <Name><First>Joe</First><Last>Public</Last></Name> <Status>U3</Status> <CrsTaken CrsCode= CS308 Semester= F1994 /> <CrsTaken CrsCode= MAT123 Semester= F1997 /> </Student> <Student StudId= s > IDREF <Name><First>Bart</First><Last>Simpson</Last></Name> <Status>U4</Status> <CrsTaken CrsCode= CS308 Semester= F1994 /> </Student> </Students> continued ID 22

23 Report Document (cont d) <Classes> <Class> <CrsCode>CS308</CrsCode> <Semester>F1994</Semester> <ClassRoster Members= s /> </Class> <Class> <CrsCode>CS308</CrsCode> <Semester>F1997</Semester> <ClassRoster Members= s /> </Class> <Class> <CrsCode>MAT123</CrsCode> <Semester>F1997</Semester> <ClassRoster Members= s s /> </Class> </Classes> continued IDREFS 23

24 Report Document cont d ID <Courses> <Course CrsCode = CS308 > <CrsName>Market Analysis</CrsName> </Course> <Course CrsCode = MAT123 > <CrsName>Market Analysis</CrsName> </Course> </Courses> </Report> 24

25 Limitations of DTDs Don t understand namespaces Very limited assortment of data types (just strings) Very weak w.r.t. consistency constraints (ID/IDREF/IDREFS only) Can t express unordered contents conveniently All element names are global: can t have one Name type for people and another for companies: <!ELEMENT Name (Last, First)> <!ELEMENT Name (#PCDATA)> both can t be in the same DTD 25

26 XML Schema Came to rectify some of the problems with DTDs Advantages: Integrated with namespaces Many built-in types User-defined types Has local element names Powerful key and referential constraints Disadvantages: Unwieldy much more complex than DTDs 26

27 XML Query Languages XPath core query language. Very limited, a glorified selection operator. Very useful, though: used in XML Schema, XSLT, XQuery, many other XML standards XSLT a functional style document transformation language. Very powerful, very complicated XQuery W3C standard. Very powerful, fairly intuitive, SQL-style SQL/XML attempt to marry SQL and XML, part of SQL:

28 Why Query XML? Need to extract parts of XML documents Need to transform documents into different forms Need to relate join parts of the same or different documents 28

29 XPath Analogous to path expressions in object-oriented languages (e.g., OQL) Extends path expressions with query facility XPath views an XML document as a tree Root of the tree is a new node, which doesn t correspond to anything in the document Internal nodes are elements Leaves are either Attributes Text nodes Comments Other things that we didn t discuss (processing instructions, ) 29

30 XPath Document Tree Root of XML tree Root of XML document 30

31 Document Corresponding to the Tree A fragment of the report document from earlier <?xml version= 1.0?> <!-- Some comment --> <Students> <Student StudId= > <Name><First>John</First><Last>Doe</Last></Name> <Status>U2</Status> <CrsTaken CrsCode= CS308 Semester= F1997 /> <CrsTaken CrsCode= MAT123 Semester= F1997 /> </Student> <Student StudId= > <Name><First>Bart</First><Last>Simpson</Last></Name> <Status>U4</Status> <CrsTaken CrsCode= CS308 Semester= F1994 /> </Student> </Students> <!-- Some other comment --> 31

32 Terminology Parent/child nodes, as usual Child nodes (that are of interest to us) are of types: text, element, attribute. Ancestor/descendant nodes as usual in trees 32

33 XPath Basics An XPath expression takes a document tree as input and returns a multi-set of nodes of the tree Expressions that start with / are absolute path expressions / /Students/Student /Student 33

34 XPath Basics An XPath expression takes a document tree as input and returns a multi-set of nodes of the tree Expressions that start with / are absolute path expressions / returns root node of XPath tree /Students/Student returns all Student-elements that are children of Students elements, which in turn must be children of the root /Student returns empty set (no such children at root) 34

35 XPath Basics cont d Current (or context node) exists during the evaluation of XPath expressions (and in other XML query languages). denotes the current node;.. denotes the parent foo/bar./foo/bar../abc/cde Expressions that don t start with / are relative (to the current node) 35

36 XPath Basics cont d Current (or context node) exists during the evaluation of XPath expressions (and in other XML query languages). denotes the current node;.. denotes the parent foo/bar returns all bar-elements that are children of foo nodes, which in turn are children of the current node./foo/bar same../abc/cde all cde e-children of abc e-children of the parent of the current node Expressions that don t start with / are relative (to the current node) 36

37 Attributes, Text, etc. Denotes an attribute StudentId /Students/Student/Name/Last/text( text( ) XPath provides means to select other document components as well 37

38 Attributes, Text, etc. Denotes an attribute StudtId returns all StudentId a-children of Student, which are e-children of Students, which are children of the root /Students/Student/Name/Last/text( text( ) returns all t- children of Last e-children of XPath provides means to select other document components as well 38

39 Overall Idea and Semantics An XPath expression is: locationstep1/locationstep2/ Location step: Axis::nodeSelector[predicate] Navigation axis: child, parent have seen This is called full syntax. We used abbreviated syntax before. Full syntax is better for describing meaning. Abbreviated syntax is better for programming. ancestor, descendant, ancestor-or-self, descendant-or-self will see later some other Node selector: node name or wildcard; e.g.,./child::student (we used./student, which is an abbreviation)./child::* any e-child (abbreviation:./*) Predicate: a selection condition; e.g., Students/Student[CourseTaken/@CrsCode = CSC343 ] 39

40 XPath Semantics The meaning of the expression locationstep1/locationstep2/ is the set of all document nodes obtained as follows: Find all nodes reachable by locationstep1 from the current node For each node N in the result, find all nodes reachable from N by locationstep2; take the union of all these nodes For each node in the result, find all nodes reachable by locationstep3, etc. The value of the path expression on a document is the set of all document nodes found after processing the last location step in the expression 40

41 Overall Idea of the Semantics cont d locationstep1/locationstep2/ means: Find all nodes specified by locationstep1 For each such node N: sfind all nodes specified by locationstep2 using N as the current node Take union For each node returned by locationstep2 do the same locationstep = axis::node[predicate] Find all nodes specified by axis::node Select only those that satisfy predicate 41

42 More on Navigation Primitives 2 nd CrsTaken child of 1 st Student child of Students: /Students/Student[1]/CrsTaken[2] All last CourseTaken elements within each Student element: /Students/Student/CrsTaken[last( )] 42

43 Wildcards Wildcards are useful when the exact structure of document is not known Descendant-or-self axis, // : allows to descend down any number of levels (including 0) //CrsTaken all CrsTaken nodes under the root Students//@Name all Name attribute nodes under the elements Students, who are children of the current node The * wildcard: * any element: any attribute: Students//@* 43

44 XPath Queries (selection predicates) Recall: Location step = Axis::nodeSelector[predicate predicate] Predicate: XPath expression = const built-in function XPath expression XPath expression built-in predicate a Boolean combination thereof Axis::nodeSelector[predicate predicate] Axis::nodeSelector but contains only the nodes that satisfy predicate Built-in predicate: special predicates for string matching, set manipulation, etc. Built-in function: large assortment of functions for string manipulation, aggregation, etc. 44

45 XPath Queries Examples Students who have taken CSC343: CSC343 ] True if : CSC343 //Student/CrsTaken/@CrsCode Complex example: //Student[Status= U3 and starts-with(.//last, A ) and contains(concat(.//@crscode, ), ESE ) and not(.//last =.//First) ] Aggregation: sum( ), count( ) //Student[sum(.//@Grade) div count(.//@grade) > 3.5] 45

46 Xpath Queries cont d Testing whether a subnode exists: //Student[CrsTaken/@Grade] students who have a grade (for some course) //Student[Name/First or CrsTaken/@Semester or Status/text() = U4 ] students who have either a first name or have taken a course in some semester or have status U4 Union operator, : //CrsTaken[@Semester= F2001 ] //Class[Semester= F1990 ] union lets us define heterogeneous collections of nodes 46

47 XQuery XML Query Language Integrates XPath with earlier proposed query languages: XQL, XML-QL SQL-style, not functional-style 2007: XQuery

48 XQuery Basics: FLOWR Expression General structure: FOR variable declarations LET variable declarations WHERE condition ORDER BY list RETURN document XQuery expression 48

49 XQuery Basics: FLOWR Expression Example: (: students who took MAT123 :) FOR $t IN doc( transcript.xml )//Transcript WHERE = MAT123 RETURN $t/student Result: comment This document on next slides <Student StudId= Name= John Doe /> <Student StudId= Name= Joe Blow /> 49

50 transcript.xml <Transcripts> <Transcript> <Student StudId= Name= John Doe /> <CrsTaken CrsCode= CS308 Semester= F1997 Grade= B /> <CrsTaken CrsCode= MAT123 Semester= F1997 Grade= B /> <CrsTaken CrsCode= EE101 Semester= F1997 Grade= A /> <CrsTaken CrsCode= CS305 Semester= F1995 Grade= A /> </Transcript> <Transcript> <Student StudId= Name= Bart Simpson /> <CrsTaken CrsCode= CS305 Semester= F1995 Grade= C /> <CrsTaken CrsCode= CS308 Semester= F1994 Grade= B /> </Transcript> cont d 50

51 transcript.xml (cont d) <Transcript> <Student StudId= Name= Joe Blow /> <CrsTaken CrsCode= CS315 Semester= S1997 Grade= A /> <CrsTaken CrsCode= CS305 Semester= S1996 Grade= A /> <CrsTaken CrsCode= MAT123 Semester= S1996 Grade= C /> </Transcript> <Transcript> <Student StudId= Name= Homer Simpson /> <CrsTaken CrsCode= EE101 Semester= F1995 Grade= B /> <CrsTaken CrsCode= CS305 Semester= S1996 Grade= A /> </Transcript> </Transcripts> 51

52 XQuery Basics (cont d) Previous query doesn t produce a well-formed XML document; the following does: <StudentList> { FOR $t IN doc( transcript.xml )//Transcript WHERE $t/crstaken/@crscode = MAT123 RETURN $t/student } </StudentList> Query inside XML 52

53 Document Restructuring with XQuery Reconstruct lists of students taking each class using the Transcript records: FOR $c IN doc( transcript.xml )//CrsTaken ORDER BY RETURN <ClassRoster > { FOR $t IN doc( transcript.xml )//Transcript WHERE $t/crstaken[@crscode = $c/@crscode = $c/@semester] ORDER BY $t/student/@studid RETURN $t/student } </ClassRoster> Query inside RETURN 53

54 Document Restructuring (cont d) Output elements have the form: <ClassRoster CrsCode= CS305 Semester= F1995 > <Student StudId= Name= John Doe /> <Student StudId= Name= Bart Simpson /> </ClassRoster> Problem: the above element will be output twice: once when $c is bound to <CrsTaken CrsCode= CS305 Semester= F1995 Grade= A /> and once when $c is bound to <CrsTaken CrsCode= CS305 Semester= F1995 Grade= C /> Bart Simpson s John Doe s Note: grades are different distinct-values( ) won t eliminate transcript records that refer to same class! 54

55 Document Restructuring cont d Solution: instead of for $c in doc( transcript.xml )//CrsTaken use for $c in doc( classes.xml )//Class Document on next slide where classes.xml lists course offerings (course code/semester) explicitly (no need to extract them from transcript records). Then $c is bound to each class exactly once, so each class roster will be output exactly once 55

56 <Classes> <Class CrsCode= CS308 Semester= F1997 > <CrsName>SE</CrsName> <Instructor>Adrian Jones</Instructor> </Class> <Class CrsCode= EE101 Semester= F1995 > <CrsName>Circuits</CrsName> <Instructor>David Jones</Instructor> </Class> <Class CrsCode= CS305 Semester= F1995 > <CrsName>Databases</CrsName> <Instructor>Mary Doe</Instructor> </Class> <Class CrsCode= CS315 Semester= S1997 > <CrsName>TP</CrsName> <Instructor>John Smyth</Instructor> </Class> <Class CrsCode= MAR123 Semester= F1997 > <CrsName>Algebra</CrsName> <Instructor>Ann White</Instructor> </Class> </Classes> 56

57 Document Restructuring (cont d) More problems: the above query will list classes with no students. Reformulation that avoids this: FOR $c IN doc( classes.xml )//Class Test that classes WHERE doc( transcripts.xml = aren t empty = $c/@semester] ORDER BY $c/@crscode RETURN <ClassRoster CrsCode = {$c/@crscode} Semester = {$c/@semester} > { FOR $t IN doc( transcript.xml )//Transcript WHERE $t/crstaken[@crscode = $c/@crscode = $c/@semester] ORDER BY $t/student/@studid RETURN $t/student } </ClassRoster> 57

58 XQuery Semantics So far the discussion was informal XQuery semantics defines what the expected result of a query is Defined analogously to the semantics of SQL 58

59 XQuery Semantics cont d Step 1: 1 Produce a list of bindings for variables The FOR clause binds each variable to a list of nodes specified by an XQuery expression. The expression can be: An XPath expression An XQuery query A function that returns a list of nodes End result of a FOR clause: Ordered list of tuples of document nodes Each tuple is a binding for the variables in the FOR clause 59

60 XQuery Semantics cont d Example (bindings): Let FOR declare $A and $B Bind $A to document nodes {v,w}; $B to {x,y,z} Then FOR clause produces the following list of bindings for $A and $B: $A/v, $B/x $A/v, $B/y $A/v, $B/z $A/w, $B/x $A/w, $B/y $A/w, $B/z 60

61 XQuery Semantics cont d Step 2: 2 filter the bindings via the WHERE clause Use each tuple binding to substitute its components for variables; retain those bindings that make WHERE true Example: WHERE $A/CrsTaken/@CrsCode = $B/Class/@CrsCode Binding: $A/w, where w = <CrsTaken CrsCode= CS308 /> $B/x, where x = <Class CrsCode= CS308 /> Then w/crstaken/@crscode = x/class/@crscode, so the WHERE condition is satisfied & binding retained 61

62 XQuery Semantics cont d Step 3: 3 Construct result For each retained tuple of bindings, instantiate the RETURN clause This creates a fragment of the output document Do this for each retained tuple of bindings in sequence 62

63 Grouping and Aggregation Does not use separate grouping operator Uses built-in aggregate functions count, avg, sum, etc. (some borrowed from XPath) 63

64 Aggregation Example Produce a list of students along with the number of courses each student took: FOR $t IN fn:doc( transcripts.xml )//Transcript, $s IN $t/student LET $c := $t/crstaken RETURN <StudentSummary StudId = {$s/@studid} Name = {$s/@name} TotalCourses = {fn:count(fn:distinct-values($c))} /> The grouping effect is achieved because $c is bound to a new set of nodes for each binding of $t 64

65 Quantification in XQuery XQuery supports explicit quantification: SOME ( ) and EVERY ( ) Example: FOR $t IN fn:doc( transcript.xml )//Transcript WHERE SOME $ct IN $t/crstaken SATISFIES $ct/@crscode = MAT123 RETURN $t/student Almost equivalent to: FOR $t IN fn:doc( transcript.xml )//Transcript, $ct IN $t/crstaken WHERE $ct/@crscode = MAT123 RETURN $t/student Not equivalent, if students can take same course twice! 65

66 Implicit Quantification Note: in SQL, variables that occur in FROM, but not SELECT are implicitly quantified with In XQuery, variables that occur in FOR, but not RETURN are similar to those in SQL. However: In XQuery variables are bound to document nodes In SQL a variable can be bound to the same value only once; identical tuples are not output twice (in theory) This is why the two queries in the previous slide are not equivalent 66

67 Quantification cont d Retrieve all classes (from classes.xml) where each student took MAT123 Hard to do in SQL (before SQL-99) because of the lack of explicit quantification FOR $c IN fn:doc(classes.xml)//class LET $g := { (: Transcript records that correspond to class $c :) FOR $t IN fn:doc( transcript.xml )//Transcript WHERE $t/crstaken/@semester = $c/@semester AND $t/crstaken/@crscode = $c/@crscode RETURN $t } WHERE EVERY $tr IN $g SATISFIES NOT fn:empty($tr[crstaken/@crscode= MAT123 ]) RETURN $c ORDER BY $c/@crscode 67

68 XQuery Functions: Example Count the number of e-children recursively: Function signature DECLARE FUNCTION countnodes($e AS element()) AS integer { } RETURN IF empty($e/*) THEN 0 ELSE sum(for $n IN $e/* RETURN countnodes($n)) + count($e/*) XQuery expression Built-in functions sum, count, empty 68

69 User-defined Functions Can define functions, even recursive ones Functions can be called from within an XQuery expression Body of function is an XQuery expression Result of expression is returned Result can be a primitive data type (integer, string), an element, a list of elements, a list of arbitrary document nodes, 69