Notes on XML and XQuery in Relational Databases

Transcription

1 xquery.txt Tue Apr 04 11:29: Notes on XML and XQuery in Relational Databases Owen Kaser March 22, Updated April 4, 2017 some code frags are untested! As usual, the idea is to give you a taste of some things, without exploring all details, but letting you do some simple things. There are likely a fair number of errors in this, though I have tested many code fragments. Corrections appreciated.

2 xquery.txt Tue Apr 04 11:29: Prerequisites Your textbook gives a quick introduction to XML and XQuery. Plus it mentions that XML can be stored and manipulated in relational databases. Make sure you understand what it says. We ll look at this with a few more details.

3 xquery.txt Tue Apr 04 11:29: Mini Review XML documents and fragments contain nested elements (delimited by tags). Elements can have attributes. <classroster> <student snum="12345"> <sname> Owen Kaser </sname> <gpa> 1.1 </gpa> <friends> <friend> <student snum="6789"/> </friend> <friend> <student snum="1111"> <gpa> 2.2 </gpa> </student></friend> </friends></student> <student snum="33333"> <sname> Newo Resak </sname> <age> 25 </age></student> </classroster>

4 xquery.txt Tue Apr 04 11:29: XML Review In previous example, we have elements: classroster, student, sname, gpa, friends, friend. tags: eg <classroster>, </classroster> and <student/> attributes: snum text: eg Newo Resak. There is a "family tree" type relationship between elements: the classroster element is the parent of two student elements. We speak of parents, children, siblings, ancestors, descendants, etc. Topmost element is called the "root". We are not told whether there is an applicable schema: we don t know whether snum must always be an integer, whether the text for age could be "really old" instead of "25", etc.

5 xquery.txt Tue Apr 04 11:29: XPath [ The W3C defines an XPath language that you can use to locate elements within a bunch of XML. It is important in other XML technologies, including XQuery. XPath lets you write expressions that return a "node" or a set of nodes. "Nodes" include elements, attributes, text and 4 other kinds. Example paths in example XML doc /classroster/student[1]//gpa start at root, which must a classroster node of its student element children, take first. any gpa elements that are descendants are returned. In this case, just two: <gpa> 2.2 </gpa> and <gpa> 1.1 </gpa>

6 xquery.txt Tue Apr 04 11:29: More XPath examples //gpa this gives the set of all gpa elements anywhere. (In this example, there are two) //@snum gives the set of all snum attributes anywhere //student//friend//@snum only those snum attributes somewhere inside a friend element, somewhere inside a student element

7 xquery.txt Tue Apr 04 11:29: Wildcards * matches any element node *@ matches any attribute node node() matches any node text() matches any text node ** possible broken note: example queries seem to put node() and text() in [] for some reason eg //friends//@* will return same thing as //friends//@snum in our example, because we don t have any other attributes

8 xquery.txt Tue Apr 04 11:29: Axes //, / seem to be shorthands axis names: child (seems like default, with /) attribute (seems prefix) descendant (seems like //) ancestor preceding (everything before here) following Eg: descendant::friends/descendant::*/attribute::snum same as //friends//@snum If current node is the first student, then following::* is be the second student (all that is left)

9 xquery.txt Tue Apr 04 11:29: Predicates Sometimes, you want to provide tests. Use [] eg. /classroster/student[@snum>10]/gpa (Both relevant snums (12345 and 33333) pass the test) //student[gpa > 1.0]

10 xquery.txt Tue Apr 04 11:29: Merging node sets The operator can form a union eg //gpa //friends

11 xquery.txt Tue Apr 04 11:29: XPath functions( There are MANY useful functions, including doc(url) - read an XML node from a file or URL eg doc("myfile.xml") count(item list), avg(number list) - count, average also max, min, sum filter, fold-left, fold-right, for-each: for "functional programming", beyond our scope has-children() - boolean head(list of items) - gives first item tail(list of items) - all but head insert-before(list of items) number() - item as a double subsequence(list, startpos, length) string-length, upper-case op:to(low,high) or use the "to" operator 1 to 5 generates sequence 1,2,3,4,5 op:is-same-node or use the "is" operator zillions of date/time functions

12 xquery.txt Tue Apr 04 11:29: XQuery XQuery is supposed to be "the SQL of XML". Another W3C technology. Uses XPath heavily. All XPath functions listed earlier can be used. A single XPath expression can be an XQuery FLWOR expressions: For-Let-Where-OrderBy-Return optional sections, in that order (well, Return is mandatory, and either For or Let) Variables with $ prefix. case sensitive Strings with or " (: This is a comment :)

13 xquery.txt Tue Apr 04 11:29: Comparisons: = vs eq : Suppose XPR is an XPath expression XPR eq 5 requires that the XPath set have exactly one value in it. Is it equal to 5? XPR = 5 can handle XPath sets with several values; if any of them are 5, you re good Also, eq would require xsd:integer for XPR but without a schema to say otherwise, XPR might have incompatible type untypedatomic. But = can handle this. Similary for > and gt, <= and le, etc.

14 xquery.txt Tue Apr 04 11:29: XQuery: for for VAR in EXPRESSION where EXPRESSION is an XPath expression or a function giving a sequence eg: for $foo in doc("myfile.xml")//friend eg: for $foo in (3 to 10)

15 xquery.txt Tue Apr 04 11:29: XQuery: nested for for $foo in (1 to 5), $bar in (2 to 6) essentially, the $bar loop is nested.

16 xquery.txt Tue Apr 04 11:29: XQuery: let Let allows an assignment, goes with := operator let $midpoint := ($start + $finish) div 2

17 xquery.txt Tue Apr 04 11:29: XQuery: where The where clause provides additional filtering, beyond what you may have done with predicates in the XPath statement. If you have "nested fors", this can be extra useful. for $x in (1 to 5), $y in (1 to 5) let $mid = ($x + $y) div 2 where $x gt $y return <owen> { $mid } </owen>

18 xquery.txt Tue Apr 04 11:29: XQuery: order by Order by does what you expect... order by $x/gpa

19 xquery.txt Tue Apr 04 11:29: XQuery: return Each ordered result then produces an XML fragment. You can use an XPath expression eg return $x/gpa Or you can generate some XML elements with varying stuff based on your variables. Note the use of {} eg return <foo><bar>{$x}</bar></foo> There is an if-then-else expression, plus all those lovely XPath functions

20 xquery.txt Tue Apr 04 11:29: Defining your own functions You can even define your own functions, though that s getting too fancy for us. For instance declare function foo ( $param1 as node()?, $param2 as xs:anyatomictype) as xs:anyatomictype* { if ($param1) then fn:data($param1) else $param2 }

21 xquery.txt Tue Apr 04 11:29: XQuery and XML in Oracle 12c see docs.oracle.com/database/121/adxdb/xdb_xquery.htm The SQL standard has a part called SQL/XML that standardizes the use of XML in SQL databases. Oracle SQL uses a data type XMLTYPE whereas SQL/XML says it should have been called XML. You can have columns of xmltype in your tables. Optionally, tell Oracle how to store the XML columns and do things to help indexing the XML. eg create table t( x xmltype);

22 xquery.txt Tue Apr 04 11:29: Getting some XML into a table There are a variety of ways to get XML. One is to use xmltype as a constructor, to convert a string into XML. insert into t values (xmltype( <classroster> <student snum="12345"> <sname> Owen Kaser </sname> <gpa> 1.1 </gpa> <friends>...stuff omitted... </classroster> )); Table t now has one row.

23 xquery.txt Tue Apr 04 11:29: Getting XML out of a table Can supposedly process the whole table as hunk of XML. In a table with an XMLType column and nothing else, Oracle docs suggest you can retrieve the "pseudocolumn" OBJECT_VALUE eg select object_value from t; Note that it didn t work for me.

24 xquery.txt Tue Apr 04 11:29: Getting XML fragments out You can process individual fragments of XML from a table using "XMLQuery". It takes an XQuery and passes in a value (t.x below) for processing select xmlquery( for $i in //student where $i/gpa > 1.5 order by $i/gpa return $i/@snum passing x returning content) from t; This returns The "passing" and "returning content" keywords are mandatory with xmlquery.

25 xquery.txt Tue Apr 04 11:29: "Passing as" You can associate a table attribute to a variable using "passing as". Below, variable $d is set to the chunk of XML in attribute x. select xmlquery( for $i in $d//student where $i/gpa > 1.5 order by $i/gpa return $i/@snum passing x as "d" returning content) from t; This returns 1111, as before.

26 xquery.txt Tue Apr 04 11:29: XMLExists and XMLcast XMLExists is useful for Boolean tests. It returns true if an XQuery expression has any results. XMLCast can turn xml into a string, discarding tags select xmlcast(x as varchar(1000)) from t where xmlexists( $y//gpa passing x as "y"); This returns a varchar version of the text portions of our data, since there was at least one gpa element. Owen Kaser Newo Resak 25

27 xquery.txt Tue Apr 04 11:29: XMLTable XMLTable converts the result of an XQuery into a relational table. Suppose that our usual table t had its usual XML column x, but also had a VARCHAR column y select y, xx.stunum, xx.gpacol from t, xmltable( /classroster//student passing t.x columns stunum int gpacol number path gpa ) xx where xx.gpacol > 1.2; For each row of t, t.x will be turned into a little table with a row for each student element in t.x, along with the student s id number and gpa (which might be null)

28 xquery.txt Tue Apr 04 11:29: Manufacturing XML yourself use XMLElement and XMLAttributes as follows select xmlelement("foo", xmlelement("contact", xmlattributes(e.employeeid as "bar"))) from employee_t e; to produce (one toplevel element per employee) <foo><contact bar=" "></contact></foo> <foo><contact bar=" "></contact></foo> <foo><contact bar=" "></contact></foo> <foo><contact bar=" "></contact></foo> <foo><contact bar="laura"></contact></foo>

29 xquery.txt Tue Apr 04 11:29: XMLAgg If you want to put the various rows into a single XML item, use XMLAgg as follows: select xmlelement("foo", (xmlagg( xmlelement("contact", xmlattributes(e.employeeid as "bar"))))) from employee_t e; which gives <foo> <contact bar=" "> </contact> <contact bar=" "></contact> <contact bar=" "></contact> <contact bar=" "></contact> <contact bar="laura"></contact> </foo>

30 xquery.txt Tue Apr 04 11:29: Summary Conventional relational databases like Oracle can handle semi-structured XML documents. XPath and XQuery give many opportunities to work with the XML information (But native XML databases might be more efficient if all you want to do is process lots of XML.)

31 xquery.txt Tue Apr 04 11:29: XML s alternative: JSON XML is rather verbose. It has a lot of technology built around it (eg, XQuery) and libraries for XML processing are available for all major languages. It has been widely used for information exchange. But its verbosity hurts. JSON (Javascript Object Notation) now challenges XML for information interchange. It s simpler. JSON s essentially a dump of a Javascript data structure. So if the JSON is received by Javascript code (eg running in a browser for a web application), it is more easily dealt with than XML.

32 xquery.txt Tue Apr 04 11:29: JSON example (our running example in JSON) Thanks to "classroster": { "student": [ { "-snum": "12345", "sname": " Owen Kaser ", "gpa": " 1.1 ", "friends": { "friend": [ { "student": { "-snum": "6789" }}, { "student": { "-snum": "1111", "gpa": " 2.2 " }}]}}, { "-snum": "33333", "sname": " Newo Resak ", "age": " 25 " }]}}

33 xquery.txt Tue Apr 04 11:29: Ease of Use Javascript code easily reads JSON into an in-memory object and processes it there, natively. Something similar using XML requires more complex processing: a "DOM parser" builds a data structure that requires tedious API calls to process. Though XML is a pain to process this way, it has nifty standardized tools like XQuery that enable higher-level processing. The JSON world wasn t as rich. But XQuery 3.1 (we use earlier versions) has support for converting JSON to/from XML. REST web services tend to prefer JSON over XML. SOAP web services require XML. SOAP can do some things that REST can t. But most people don t need to do these complicated things.

34 xquery.txt Tue Apr 04 11:29: SOAP/XML vs REST/JSON example From keithba.net/simplicity-and-utility-or-why-soap-lost POST /customers HTTP/1.1 Host: Content-Type: application/soap+xml; charset=utf-8 <?xml version="1.0"?> <soap:envelope xmlns:soap=" <soap:body> <m:getcustomer xmlns:m=" <m:customerid>43456</m:customerid> </m:getcustomer> </soap:body> </soap:envelope> for SOAP, vs for REST GET /customers/43456 HTTP/1.1 Host:

35 xquery.txt Tue Apr 04 11:29: SOAP/XML vs REST/JSON example, Data returned, first for SOAP HTTP/ OK Content-Type: application/soap+xml; charset=utf-8 <?xml version= 1.0?> <env:envelope xmlns:env=" > <env:body> <m:getcustomerresponse xmlns:m=" <m:customer>foobar Quux, inc</m:customer> </m:getcustomerresponse> </env:body> </env:envelope> then for REST HTTP/ OK Content-Type: application/json; charset=utf-8 { Customer : Foobar Quux, inc }