Next Generation Query and Transformation Standards. Priscilla Walmsley Managing Director, Datypic

Transcription

1 Next Generation Query and Transformation Standards Priscilla Walmsley Managing Director, Datypic 1

2 Agenda The query and transformation landscape Querying XML with XQuery Transforming XML with XSLT Shared components Decision points 2005 Datypic Slide 2

3 Querying and Transforming XML Querying: Extracting data of interest Transformation: Changing the structure of data Sometimes it's hard to tell them apart Give me just the b elements, but call them x in the results Change all the b elements to x elements, and ignore the rest 2005 Datypic Slide 3

4 W3C Standards for Querying/Transformation FLWOR Expressions XML Constructors Query Prolog User-Defined Functions Conditional Expressions Arithmetic Expressions Quantified Expressions Built-In Functions & Operators Data Model XPath XQuery XSLT 2.0 XPath 1.0 Path Expressions Comparison Expressions Some Built-In Functions Stylesheets Templates XML Constructors User-Defined Functions 2005 Datypic Slide 4

5 Querying XML with XQuery 5

6 XQuery 1.0 for $i1 in for $i2 in doc("input2.xml")//product where = order by $i1 return <dep name="{$i1}" XML Input 1 XML Input 2 parse parse XQuery Processor analyze and evaluate serialize (or pass on) XML XML Output Output 2005 Datypic Slide 6

7 XML Input Could be data that is: a textual XML document on a file system retrieved from a Web service stored in an XML database stored in a relational database created in memory by program code Can take the form of: a single XML document a collection of several documents a fragment of a document (e.g. sequence of elements) 2005 Datypic Slide 7

8 XQuery 1.0 Capabilities selecting elements/attributes from XML input documents for $i1 in doc("order.xml")//item for $i2 in doc("catalog.xml")//product where = $i2/prodnum order by $i1/quantity return <item name="{$i2/prodname}" saleprice="{min($i2/price) - $i2/discount}"/> 2005 Datypic Slide 8

9 XQuery 1.0 Capabilities joining data from multiple sources for $i1 in doc("order.xml")//item for $i2 in doc("catalog.xml")//product where = $i2/prodnum order by $i1/quantity return <item name="{$i2/prodname}" saleprice="{min($i2/price) - $i2/discount}"/> 2005 Datypic Slide 9

10 XQuery 1.0 Capabilities adding new elements/attributes to results for $i1 in doc("order.xml")//item for $i2 in doc("catalog.xml")//product where = $i2/prodnum order by $i1/quantity return <item name="{$i2/prodname}" saleprice="{min($i2/price) - $i2/discount}"/> 2005 Datypic Slide 10

11 XQuery 1.0 Capabilities performing calculations for $i1 in doc("order.xml")//item for $i2 in doc("catalog.xml")//product where = $i2/prodnum order by $i1/quantity return <item name="{$i2/prodname}" saleprice="{min($i2/price) - $i2/discount}"/> 2005 Datypic Slide 11

12 XQuery 1.0 Capabilities sorting results for $i1 in doc("order.xml")//item for $i2 in doc("catalog.xml")//product where = $i2/prodnum order by $i1/quantity return <item name="{$i2/prodname}" saleprice="{min($i2/price) - $i2/discount}"/> 2005 Datypic Slide 12

13 XQuery Use Cases 13

14 Search and Browse What hotels in New York allow pets and have Internet access? Happy User Built-In User Interface Built-In XQuery Processor Custom User Interface Semi-Structured XML Content (Poetry Manuscripts, Medical Journals, Hotel Reviews) "Native" XML DBMS e.g. MarkLogic, Berkeley DB, exist 2005 Datypic Slide 14

15 "XML-izing" Data for Web Services What is the status of my order? Happy User Order Inquiry Web Service Built-In XQuery Front-End Structured Data (Orders, Product Prices, Customer Information) Relational DBMS e.g. SQL Server, Oracle, DB Datypic Slide 15

16 Integrating Disparate Data Sources DataDirect Technologies 2005 Datypic Slide 16

17 Anything, really... Anywhere in application code you would currently use XPath, or XSLT, or DOM, e.g.: to narrow down results returned from a Web service in a pipeline process to split or subset an XML document to manipulate or create a configuration file stored as XML 2005 Datypic Slide 17

18 XQuery Features 18

19 Features of XQuery Compact syntax Typing and schema support Reusable function libraries Designed with today's XML in mind 2005 Datypic Slide 19

20 Compact, Intuitive Syntax Easy to learn and use Less verbose than XSLT but much more powerful than straight XPath Does not require hard-core programming background Ideal for embedding into programming languages 2005 Datypic Slide 20

21 Embedding in Java XQJ: XQuery API for Java proposed Java standard for invoking queries, and processing the results the "JDBC of XML" XQExpression expr = conn.createexpression(); String qy = "for $p in doc('cat.xml')//product return ($p/name)"; XQResultSequence result = expr.executequery(qy); while (result.next()) { String str = result.getstring(); System.out.println("Product name: " + str); } result.close(); expr.close(); conn.close(); 2005 Datypic Slide 21

22 Typing and Schema Support Typing allows for identification of query errors Optional schema support can associate a schema with a query or input document the schema defines the rules for the input or output XML names of elements/attributes hierarchical structure number of occurrences data types 2005 Datypic Slide 22

23 Benefits of Using Schemas Better identification of static errors allows discovery of errors in the query that were not otherwise apparent especially important when new versions of the input XML vocabulary come along Query optimization Validity of query inputs and results makes them more predictable Special processing based on type 2005 Datypic Slide 23

24 Using Schemas to Catch Static Errors import schema default element namespace " at " for $prod in doc("cat.xml")/produt order by $prod/name/number return $prod/name + 1 misspelling type error: name is declared to be of type xs:string, so cannot be used in an add operation invalid path; name will never have number child 2005 Datypic Slide 24

25 Reusable Function Libraries Portable, reusable, shareable Can provide a set of standard queries on a standard XML vocabulary As vocabulary changes, function libraries can be recompiled and/or versioned module namespace dty = " declare function dty:orderstatus($num as xs:string?) as element(order)* {... }; declare function dty:cancelorder($num as xs:string?) as xs:boolean {... }; 2005 Datypic Slide 25

26 Designed with Today's XML in Mind Intuitive, designed-in support for: namespaces construction of new elements/attributes data types whitespace handling etc. Much less awkward than, e.g., DOM manipulation 2005 Datypic Slide 26

27 Transforming XML with XSLT 27

28 Typical XSLT Use Cases Transform content into presentation XML to HTML, XML to XSL-FO General purpose XML to XML transforms (data manipulation) B2B EAI Transform XML to other formats (text, CSV, etc.) 2005 Datypic Slide 28

29 XSLT (Look familiar?) <xsl:template match="order"> <xsl:for-each select="item"> <li>item number <xsl:value-of </xsl:for-each> </xsl:template> XML Input 1 XML Input 2 parse parse XSLT Processor analyze and evaluate serialize (or pass on) XML XML Output Output 2005 Datypic Slide 29

30 XSLT What's New? Grouping Multiple result documents Temporary result trees XPath 2.0 enhancements more powerful syntax more built-in functions Schema support and type system 2005 Datypic Slide 30

31 Schema Support and Type System Same typing/schema features as XQuery Special processing based on type: <xsl:template match="element(*,usaddresstype)">... <xsl:value-of select="city"/> <xsl:value-of select="zipcode"/> </xsl:template> <xsl:template match="element(*,ukaddresstype)">... <xsl:value-of select="postcode"/> <xsl:value-of select="city"/> </xsl:template> 2005 Datypic Slide 31

32 XSLT Conveniences (not present in XQuery) Highly flexible recursive processing allows "Push" approach Grouping syntax is more explicit = easier Formatting of dates and numbers format-date, format-number Advanced string manipulation analyze-string Ability to customize/override stylesheets 2005 Datypic Slide 32

33 Pull vs. Push Approaches Pull go get element X and do this with it next, go get element Y and do this with it Push get the root element if it happens to be X, do this with it. if it happens to be Y, do this with it. if it's anything else, skip it. next, go get its children and repeat 2005 Datypic Slide 33

34 Pull Approach Pulling the information from the input document using hardcoded paths to specific locations Requires a predictable document structure <xsl:template match="order"> <xsl:for-each select="item"> <li>item #<xsl:value-of select="@num"/></li> </xsl:for-each> </xsl:template> 2005 Datypic Slide 34

35 Push Approach Traversing a document, taking each element as it comes, then deciding what to do with it Useful when the structure of the input file is not known, or is highly flexible Flexible but not optimized Very difficult to do in XQuery 2005 Datypic Slide 35

36 Sample Stylesheet in "Push" Style <xsl:template match="order"> <xsl:apply-templates select="*"/> </xsl:template> <xsl:template match="item"> <xsl:apply-templates </xsl:template> <xsl:template <li>item # <xsl:value-of select="."/></li> </xsl:template> 2005 Datypic Slide 36

37 XQuery and XSLT: Shared Components 37

38 Shared Components XPath 2.0 Built-in functions Data model 2005 Datypic Slide 38

39 XPath 2.0 Full compatibility across XQuery and XSLT same syntax same expression will always return the same value Much more than just path expressions for $a in fn:distinct-values(/bib/book/author) return ($a, /bib/book[author = $a]/title) some $emp in /emps/employee satisfies ($emp/bonus > 0.25 * $emp/salary) 2005 Datypic Slide 39

40 Over 100 Built-In Functions: A Sample String-related substring, contains, matches, tokenize Date-related current-date, month-from-date Number-related round, avg, sum, ceiling Sequence-related index-of, insert-before, reverse Document- and URI-related collection, doc, root, base-uri 2005 Datypic Slide 40

41 XQuery/XPath Data Model 2005 Datypic Slide 41

42 XQuery vs. XSLT: Decision Factors 42

43 XQuery vs. XSLT: Decision Factors Use case Availability of relevant implementations Performance Programming style 2005 Datypic Slide 43

44 Use Case Use XSLT if: your documents are highly variable your transformation is presentation-oriented your processing is heavily recursive Use XQuery if: you are selecting a small subset of a collection of XML data you are joining data from multiple sources your documents are predictable in structure, or variations are not relevant to your searches 2005 Datypic Slide 44

45 Availability of Relevant Implementations XQuery XML DBMSs: MarkLogic, Sleepycat Berkeley DB, X-Hive, exist Relational DBMSs: Oracle, SQL Server, DB2 Standalone: Saxon XML Editors: Stylus Studio, XMLSpy, Oxygen XSLT 2.0 Standalone: Saxon XML Editors: Stylus Studio, XMLSpy 2005 Datypic Slide 45

46 Performance XQuery implementations tend to be optimized for: XML stored in a database predictable document structures that can be indexed XSLT implementations tend to be optimized for: transforming an entire document that can be loaded into memory More driven by use cases than limitations of languages 2005 Datypic Slide 46

47 Programming Style XSLT recursive template language difficult for some developers to grasp verbosity can be irritating however, many users loves it XQuery appealing to SQL users probably easier for newcomers 2005 Datypic Slide 47

48 Conclusions XQuery and XSLT 2.0 are coming of age They overlap in capabilities... but differ in use cases and sweet spots Both take XML manipulation to a new level in terms of: power flexibility production-readiness 2005 Datypic Slide 48

49 Resources Detailed technical comparison of XQuery and XSLT 2.0 Michael Kay's paper from XTech 05: edings/xtech05/papers/ / XQuery implementations Datypic Slide 49

50 Learning XQuery My tutorial on XQuery: Definitive XQuery By Priscilla Walmsley Coming in Datypic Slide 50

51 Thank you for your interest. For more information please contact me at: Website: 51