Querying and Transforming XML Data Chapter5 Contents

Transcription

1 Contents Outline of Lecture Sources Motivation Querying and Transforming XML Data Tree Model of XML Data XML Hierarchy What is XPath? XPath - Selecting Branches Accessing Attributes in XPath Functions in XPath More XPath Features Stylesheet Languages : CSS and XSL XSL XSLT - XSL Transformations How does it Work? XSLT Templates Creating XML Output Structural Recursion Sorting in XSLT XML Query (XQuery) Summary: FOR v.s. LET Joins Changing Nesting Structure A Piece of XML Example of Nested Elements Sorting in XQuery Functions and Other XQuery Features Some Xquery references

2 Application Program Interface Parser API: SAX Parser API: DOM DOM Parser Processing Model XML Processing: XSLT XSLT processing model XML Processing Toolkits XML and databases Product categories Exercises Some other XML formats Reference Material

3 152

4 Outline of Lecture Querying & Transformation XPath XSLT XQuery Application Program Interfaces XML and Database Objective Introduce some of the technologies used for querying and transforming XML documents. Sources Database System Concepts- Siberschatz Database Management Systems Ramakrishnan + Based on slides by Dan Suciu from University of Washington Ian GRAHAM - IT Strategy, IBS, Technology and Solutions, BMO Financial Group Motivation Given the increasing number of applications that use XML to exchange, mediate, and store data, tools for effective management of XML data are becoming increasingly important. In particular, tools for querying and transformation of XML data are essential to extract information from large bodies of XML data, and to convert data between different representations (Schemas) in XML. 153

5 Querying and Transforming XML Data Translation of information from one XML schema to another and Querying on XML data are closely related, and handled by the same tools Standard XML querying/translation languages o Xpath: Simple language consisting of path expressions o XSLT: Simple language designed for translation from XML to XML and XML to HTML o XQuery: An XML query language with a rich set of features Wide variety of other languages have been proposed, and some served as basis for the Xquery standard o XML-QL, o Quilt, o XQL, Tree Model of XML Data Query and transformation languages are based on a tree model of XML data An XML document is modeled as a tree, with nodes corresponding to elements and attributes o Element nodes have children nodes, which can be attributes or subelements o Text in an element is modeled as a text node child of the element 154

6 o Children of a node are ordered according to their order in the XML document o Element and attribute nodes (except for the root node) have a single parent, which is an element node o The root node has a single child, which is the root element of the document We use the terminology of nodes, children, parent, siblings, ancestor, descendant, etc., which should be interpreted in the above tree model of XML data XML Hierarchy 155

7 Bookstore example What is XPath? XPath is a major element in the W3C XSLT standard. Without XPath knowledge you will not be able to create XSLT documents. o XPath is a syntax for defining parts of an XML document o XPath uses paths to define XML elements o XPath defines a library of standard functions 156

8 XPath W3C specification: XPath is used to address (select) parts of documents using path expressions A path expression is a sequence of steps separated by / o Think of file names in a directory hierarchy Result of path expression: set of values that along with their containing elements/attributes match the specified path 157

9 XPath - Selecting Branches By using square brackets in an XPath expression you can specify an element further. Selection predicates may follow any step in a path, in [ ] E.g., o /bank-2/account[1] selects the first account child element of the bank-2 element o /bank-2/account[last()] selects the last account child element. Note there is no function first() o /bank-2/account[balance] 158

10 selects all the account elements of the bank-2 element that have a balance element: o /bank-2/account[balance>400] The initial / denotes root of the document (above the top-level tag) Path expressions are evaluated left to right o Each step operates on the set of instances produced by the previous step Accessing Attributes in XPath Attributes are accessed o E.g. /bank-2/account[balance > 400]/@account-number returns the account numbers of those accounts with balance > 400 o IDREF attributes are not dereferenced automatically 159

11 Functions in XPath XPath provides several functions o The function count() at the end of a path counts the number of elements in the set generated by the path o Also function for testing position (1, 2,..) of node w.r.t. siblings Boolean connectives and and or and function not() can be used in predicates IDREFs can be referenced using function id() o id() can also be applied to sets of references such as IDREFS and even to strings containing multiple references separated by blanks o E.g. /bank-2/account/id(@owner) 160

12 returns all customers referred to from the owner s attribute of account elements More XPath Features Operator used to implement union o E.g. /bank-2/account/id(@owner) /bank-2/loan/id(@borrower) gives customers with either accounts or loans However, cannot be nested inside other operators. If the path starts with a slash ( / ) it represents an absolute path to an element! If the path starts with two slashes ( // ) then all elements in the document that fulfill the criteria will be selected (even if they are at different levels in the XML tree)! 161

13 // can be used to skip multiple levels of nodes o E.g. /bank-2//name finds any name element anywhere under the /bank-2 element, regardless of the element in which it is contained. A step in the path can go to: parents, siblings, ancestors and descendants of the nodes generated by the previous step, not just to the children o //, described above, is a short from for specifying all descendants o.. specifies the parent Stylesheet Languages : CSS and XSL A stylesheet stores formatting options for a document, usually separately from document o E.g. HTML style sheet may specify font colors and sizes for headings, etc CSS - HTML Style Sheets o HTML uses predefined tags and the meanings of tags are well understood. o The <table> element defines a table and a browser knows how to display it. o Adding styles to HTML elements is also simple. Telling a browser to display an element in a special font or color, is easily done with CSS. 162

14 XSL o XSL stands for extensible Stylesheet Language. o The World Wide Web Consortium (W3C) started to develop XSL because there was a need for an XML based Stylesheet Language. XSL XSL - XML Style Sheets o XML does not use predefined tags (we can use any tags we like) and the meanings of these tags are not well understood. o The <table> could mean an HTML table or a piece of furniture, and a browser does not know how to display it. o There must be something in addition to the XML document that describes how the document should be displayed; and that is XSL! XSL - More Than a Style Sheet Language The XML Stylesheet Language (XSL) was originally designed for generating HTML from XML W3C specification: XSL consists of three parts: o XSLT is a language for transforming XML documents o XPath is a language for defining parts of an XML document o XSL-FO is a language for formatting XML documents Think of XSL as set of languages that can o transform XML into XHTML, 163

15 o filter and sort XML data, o define parts of an XML document, o format XML data based on the data value, like displaying negative numbers in red, and o output XML data to different medias, like screens, paper, or voice. XSLT - XSL Transformations XSLT is a general-purpose transformation language o Can translate XML to XML, and XML to HTML XSLT transformations are expressed using rules called templates o Templates combine selection using XPath with construction of results XSLT is the most important part of the XSL Standards. Normally XSLT transforms each XML element into an (X)HTML element. XSLT can also o add new elements into the output file, or o remove elements o rearrange and sort elements, and o test and make decisions about which elements to display, and o a lot more. 164

16 A common way to describe the transformation process is to say that XSLT transforms an XML source tree into an XML result tree. How does it Work? XSLT uses XPath to define the matching patterns for transformations. In the transformation process, XSLT uses XPath to define parts of the source document that match one or more predefined templates. When a match is found, XSLT will transform the matching part of the source document into the result document. The parts of the source document that do not match a template will end up unmodified in the result document 165

17 166

18 167

19 XSLT- Filtering the Output We can filter the output from an XML file by adding a criterion to the select attribute in the <xsl:for-each> element. <xsl:for-each select="catalog/cd[artist='bob Dylan']"> XSLT IF o Legal filter operators are: o = (equal) o!= (not equal) o < less than o > greater than The <xsl:if> element contains a template that will be applied only if a specified condition is true. Where to put the IF condition o To put a conditional if test against the content of the file, simply add an <xsl:if> element to your XSL document inside a <xsl:foreach> element like this: <xsl:for-each select="catalog/cd"> <xsl:if test="price > 10"> some output... </xsl:if> o The code above only selects the <cd> elements where the <price> element of the cd is higher than

20 XSLT Templates Example of XSLT template with match and select part <xsl:template match= /bank-2/customer > <xsl:value-of select= customer-name /> </xsl:template> <xsl:template match= * /> The match attribute of xsl:template specifies a pattern in XPath Elements in the XML document matching the pattern are processed by the actions within the xsl:template element o xsl:value-of selects (outputs) specified values (here, customername) For elements that do not match any template o Attributes and text contents are output as is o Templates are recursively applied on subelements The <xsl:template match= * /> template matches all elements that do not match any other template o Used to ensure that their contents do not get output. If an element matches several templates, only one is used o We assume only one template matches any element Creating XML Output Any text or tag in the XSL stylesheet that is not in the xsl namespace is output as is 169

21 E.g. to wrap results in new XML elements. <xsl:template match= /bank-2/customer > <customer> <xsl:value-of select= customer-name /> </customer> </xsl:template> <xsl:template match= * /> o Example output: <customer> Joe </customer> <customer> Mary </customer> Creating XML attribute o E.g. cannot create an attribute for <customer> in the previous example by directly using xsl:value-of o XSLT provides a construct xsl:attribute to create an attribute for an element xsl:attribute adds attribute to the preceding element E.g. <customer> <xsl:attribute name= customer-id > <xsl:value-of select = customer-id /> </xsl:attribute> </customer> results in output of the form 170

22 <customer customer-id=. >. xsl:element is used to create output elements with computed names <xsl:apply-templates> element The <xsl:apply-templates> element applies a template to the current element or to the current element's child nodes. Structural Recursion Sorting in XSLT Using an xsl:sort directive inside a template causes all elements matching the template to be sorted o Sorting is done before applying other templates E.g. 171

23 XML Query (XQuery) The mission of the XML Query project is to provide flexible query facilities to extract data from real and virtual documents on the World Wide Web. Therefore finally providing the needed interaction between the web world and the database world. Ultimately, collections of XML files will be accessed like databases. What is XQuery? XQuery is a language for querying XML data. XQuery is built on XPath expressions XQuery for XML is like SQL for databases XQuery is a language for finding and extracting (querying) data from XML documents. Here is an example of a question that XQuery could solve: 172

24 o "Select all CD records with a price less than $10 from the CD collection stored in the XML document called cd_catalog.xml XQuery and XPath o XQuery 1.0 and XPath 2.0 shares the same data model, the same functions, and the same syntax. XSLT vs. XQuery XSLT 1.0 is a transforming language and is single document based. XQuery 1.0 is more a querying language, is multiple documents based, and is more suitable for big XML source documents. XQuery W3C specification: XQuery is a general purpose query language for XML data XQuery is derived from the Quilt query language, which itself borrows from SQL, XQL and XML-QL XQuery uses a for let where Orderby return syntax for SQL from where SQL where return SQL select let allows temporary variables, and has no equivalent in SQL 173

25 Summary: FOR-LET-WHERE-ORDERBY-RETURN = FLWOR XQuery FOR $x in expr o -- binds $x to each value in the list expr o Binds node variables iteration LET $x = expr o -- binds $x to the entire list expr o Useful for common subexpressions and for aggregations o Binds collection variables one value 174

26 FOR v.s. LET FLWR Syntax in XQuery For clause uses XPath expressions, and variable in for clause ranges over values in the set returned by XPath Simple FLWR expression in XQuery o find all accounts with balance > 400, with each result enclosed in an <account-number>.. </account-number> tag for $x in /bank-2/account let $ac-no := $x/@account-number where $x/balance > 400 return <account-number> $ac-no </account-number> Let clause not really needed in this query, and selection can be done in Xpath o Query can be written as: for return $x in /bank-2/account[balance>400] <account-number> $X/@account-number </account-number> 175

27 Functions The function distinct( ) can be used to remove duplicates in path expression results The function document(name) returns root of named document o E.g. document( bank-2.xml )/bank-2/account Aggregate functions such as sum( ) and count( ) can be applied to path expression results XQuery 176

28 Result: <result> <author>jones</author> <title> abc </title> <title> def </title> </result> <result> <author> Smith </author> <title> ghi </title> </result> count = a (aggregate) function that returns the number of elms Find books whose price is larger than average: 177

29 Joins Joins are specified in a manner very similar to SQL for $a in /bank/account, $c in /bank/customer, $d in /bank/depositor where $a/account-number = $d/account-number and $c/customer-name = $d/customer-name return <cust-acct> $c $a </cust-acct> The same query can be expressed with the selections specified as XPath selections: for $a in /bank/account $c in /bank/customer $d in /bank/depositor[ account-number =$a/account-number and customer-name = $c/customer-name] return <cust-acct> $c $a</cust-acct> 178

30 Changing Nesting Structure The following query converts data from the flat structure for bank information into the nested structure used in bank-1 <bank-1> for $c in /bank/customer return <customer> $c/* </bank-1> for $d in /bank/depositor[customer-name = $c/customer-name], $a in /bank/account[account-number=$d/account-number] return $a/* </customer> $c/* denotes all the children of the node to which $c is bound, without the enclosing top-level tag 179

31 A Piece of XML Example of Nested Elements 180

32 Sorting in XQuery Sortby clause can be used at the end of any expression. E.g. to return customers sorted by name for $c in /bank/customer return <customer> $c/* </customer> sortby(name) Can sort at multiple levels of nesting (sort by customer-name, and by account-number within each customer) <bank-1> for $c in /bank/customer return <customer> $c/* for $d in /bank/depositor[customer-name=$c/customer-name], $a in /bank/account[account-number=$d/account-number] return <account> $a/* </account> sortby(account-number) </customer> sortby(customer-name) </bank-1> Functions and Other XQuery Features User defined functions with the type system of XMLSchema function balances(xsd:string $c) returns list(xsd:numeric) { for $d in /bank/depositor[customer-name = $c], $a in /bank/account[account-number=$d/account-number] return $a/balance } 181

33 Path Expressions /bib/paper[2]/author[1] /bib//author paper[author/lastname= Smith"] /bib/(paper book)/title If-Then-Else Some Xquery references Book: Xquery, Priscilla Walmsley 1st edition, 2007, O'Reilly Media, Inc

34 Application Program Interface The design goals of XML include "It shall be easy to write programs which process XML documents. Despite this fact, the XML specification contains almost no information about how programmers might go about doing such processing. A variety of APIs for accessing XML have been developed and used, and some have been standardized Existing APIs for XML processing tend to fall into these categories: o Stream-oriented APIs accessible from a programming language, for example SAX and StAX. o Tree-traversal APIs accessible from a programming language, for example DOM. o XML data binding, which provides an automated translation between an XML document and programming-language objects. o Declarative transformation languages such as XSLT and XQuery. Stream-oriented facilities require less memory and, for certain tasks which are based on a linear traversal of an XML document, are faster and simpler than other alternatives. Tree-traversal and data-binding APIs typically require the use of much more memory, but are often found more convenient for use by programmers; some include declarative retrieval of document components via the use of XPath expressions. XSLT is designed for declarative description of XML document transformations, and has been widely implemented both in server-side packages and Web browsers. XQuery overlaps XSLT in its 183

35 functionality, but is designed more for searching of large XML databases. Lots of XML parsers and interface software available o Unix, Linux, Windows 2000/XP, Z/OS, etc SAX-based parsers are fast (often as fast as you can stream data) DOM based parsers are slower, more memory intensive (create inmemory version of entire document Validating can be much slower than non-validating There are lots of APIs,and lots of parsers. Most come in commercial suites XML parsers.. o Read in XML data, checks for syntactic (and possibly DTD/Schema) constraints, and makes data available to an application. Here are some 'generic' parser APIs SAX Simple API to XML (event-based) DOM Document Object Model (object/tree based) Parser API: SAX SAX: Simple API for XML o o An event-based interface (a push parser API) o Parser reports events whenever it sees a tag/attribute/text node/unresolved external entity/other (driven by input stream) o Programmer attaches event handlers to handle the event 184

36 Advantages o Simple to use o Very fast (not doing very much before you get the tags and data) o Low memory footprint (doesn t read an XML document entirely into memory) Disadvantages o Not doing very much for you -- you have to do everything yourself o Not useful if you have to dynamically modify the document once it s in memory (since you ll have to do all the work to put it in memory yourself!) SAX - Simple API for XML A SAX parser generates events o at the start and end of a document, o at the start and end of an element, o when it finds characters inside an element, and at several other points. User writes code that handles each event, and decides what to do with the information from the parser SAX parsing is unidirectional; previously parsed data cannot be re-read without starting the parsing operation again SAX (Simple API for XML) is a sequential access parser API for XML. SAX provides a mechanism for reading data from an XML document. 185

37 Parser API: DOM DOM: Document Object Model o o An object-based interface o Parser generates an in-memory tree corresponding to the document o DOM interface defines methods for accessing and modifying the tree Advantages o Very useful for dynamic modification of, access to the tree o Useful for querying (I.e. looking for data) that depends on the tree structure [element.childnode("2").getattributevalue("boobie")] o Same interface for many programming languages (C++, Java,...) Disadvantages o Can be slow (needs to produce the tree), and may need lots of memory o DOM programming interface is a bit awkward, not terribly object oriented If you ve ever used JavaScript to write Web pages scripts, then you ve likely worked with the DOM the DOM was based on dynamic HTML and dynamic scripting in Netscape Navigator and Internet Explorer browsers. The XML DOM is simply a generalization of the DOM originally designed for HTML. 186

38 DOM Parser Processing Model The parser generates the DOM tree when the document is read in. The DOM interface then lets the application access the tree. Typically the parser can operate in synchronous or asynchronous mode that is, it can choose whether or not to block access to the document object until the incoming data is completely parsed. DOM- Application Program Interface Set of interfaces for an application that reads an XML file into memory and stores it as a tree structure 187

39 As a W3C specification, the objective for the XML DOM has been to provide a standard programming interface to a wide variety of applications. The XML DOM is designed to be used with any programming language and any operating system. With the XML DOM, a programmer can create an XML document, navigate its structure, and add, modify, or delete its elements o XML data is parsed into a tree representation o Variety of functions provided for traversing the DOM tree o E.g.: Java DOM API provides Node class with methods getparentnode( ), getfirstchild( ), getnextsibling( ) getattribute( ), getelementsbytagname( ), getdata( ) (for text node) o Also provides functions for updating DOM tree XML Processing: XSLT XSLT extensible Stylesheet Language -- Transformations o 188

40 o An XML language for processing/transforming XML o Does tree transformations -- takes XML and an XSLT style sheet as input, and produces a new XML document with a different structure Advantages o Very useful for tree transformations -- much easier than DOM or SAX for this purpose o Can be used to query a document (XSLT pulls out the part you want) Disadvantages o Can be slow for large documents or stylesheets o Can be difficult to debug stylesheets (poor error detection; much better if you use schemas) XSLT is a very important tool for XML processing, particularly for converting XML data into another form of XML, or into plain text. Many XML-based content management applications of Web page generation tools use XSLT to do the conversion of page generation. 189

41 XSLT processing model An XSLT processor uses DOM-like interfaces to the XML data and XSLT style sheet. Typically, however, the two are stored differently, as they serve very different purposes. XML Processing Toolkits Lots of them Java o JAXP ( ) dom4j ( ).NET ( part of.net framework) others 190

42 o Provide DOM, SAX, (JDOM) interfaces, plus lots of other useful tools in a standardized way (loading parsers, performing XSLT transformations, etc.) XML and databases An XML database is a software system that allows data to be stored in XML format. This data can then be queried, exported and serialized into the desired format. Two major classes of XML database exist: o XML-enabled: these map all XML to a traditional database (such as a relational database), accepting XML as input and rendering XML as output. This term implies that the database does the conversion itself (as opposed to relying on middleware). o Native XML (NXD): the internal model of such databases depends on XML and uses XML documents as the fundamental unit of storage, which are, however, not necessarily stored in the form of text files. XML documents fall into two broad categories: data-centric and document-centric. Data-centric documents are those where XML is used as a data transport. They include sales orders, patient records, and scientific data. Their physical structure -- the order of sibling elements, whether data is stored in attributes or PCDATA-only elements, whether entities are used -- is often unimportant. A special case of data-centric documents is dynamic Web pages, such as online catalogs and address lists, which are constructed from known, regular sets of data. 191

43 Document-centric documents are those in which XML is used for its SGML-like capabilities, such as in user's manuals, static Web pages, and marketing brochures. They are characterized by irregular structure and mixed content and their physical structure is important. To store and retrieve the data in data-centric documents, what kind of software you need will depend on how well structured your data is. For highly structured data, such as the white pages in a telephone book, you will need an XML-enabled database that is tuned for data storage, such as a relational or object-oriented database, and some sort of data transfer software. This may be built in to the database (in which case the database is said to be XML-enabled) or might be third-party software, such as middleware, data integration software, or a Web application server. If your data is semi-structured, such as the yellow pages in a telephone book or health data, you have two choices. You can try to fit your data into a well-structured database, such as a relational database, or you can store it in a native XML database, which is designed to handle semistructured data. In addition, wrappers can treat an XML document as a source of relational data. To store and retrieve document-centric documents, you will need a native XML database or content (document) management system. (Some XML-enabled databases provide native storage as well.) Content management systems generally have additional functionality, such as editors, version control, and workflow control. Although content management systems often use a native XML database for storage, this is hidden from the user. 192

44 Product categories Middleware: Software you call from your application to transfer data between XML documents and databases. For data-centric applications. IDEs and Editors: Software designed to help you write XML applications or edit XML documents. For data-centric applications. Data Integration Software: Standalone servers designed to transfer data between multiple data sources, including XML and databases. For data-centric applications. XML-Enabled Databases: Databases with extensions for transferring data between XML documents and themselves. Primarily for datacentric applications. Native XML Databases: Databases that store XML in "native" form, generally as some variant of the DOM mapped to an underlying data store. For data- and document-centric applications. Web Application Servers: Software that builds database-driven XML documents for access over the Web. For data-centric applications. Wrappers: Software that treats XML documents as a source of relational data. These products typically query XML documents using SQL. For data-centric applications. Content (Document) Management Systems: Applications built on top of native XML databases and/or the file system for content/document management. Include features such as check-in/check-out, versioning, and editors. For document-centric applications. XML Query Engines: Standalone engines that can query XML documents. For data- and document-centric applications. 193

45 XML Data Binding: Products that can bind XML documents to objects. Some of these can also store/retrieve objects from the database. For data-centric applications Except for content management systems, you will need to write code to integrate these products with your applications, although data integration software and Web application servers generally require more scripting or GUI work, and less code, than the other categories. You will need to configure content management systems, which may be a non-trivial task in itself. Exercises Practical experience with some of the products and technologies mentioned during the XML sessions. Useful XML Database references o Introductory article o XML and databases o Products list o Docs / resource list Some other XML formats MathML for Mathematics Chemical Markup Language (CML) for Chemistry Astronomical Markup Language (AML) for Astronomy 194

46 Bioinformatic Sequence Markup Language (BSML) for the human genome project Extensible Scientific Interchange Language (XSIL) DDI for Social Science Data Conclusions Introduced some of the technologies and products used for querying and transforming and storing XML documents. XML is now achieving momentum The scientific data management community should be at the forefront of its use. o advantages of widely available tools o advantages in integration o advantages in information management Reference Material XML FAQ, XML in 10 points, W3C Communications Team, Data Management for XML Research Directions More on Data Management for XML 195

47 Describing and Manipulating XML Data, W3C on XML, 196