Introduction & Motivation. Course Outline. Arrangements. Presentation vs Structure. Markup and Markup Language. Main Topic: Two XML Query Models

Size: px

Start display at page:

Download "Introduction & Motivation. Course Outline. Arrangements. Presentation vs Structure. Markup and Markup Language. Main Topic: Two XML Query Models"

Ruby Lawson
6 years ago
Views:

Querying XML Documents and Data CBU Summer School 13.8. - 20.8.2007 (2 ECTS) Prof. Pekka Kilpeläinen inen Univ of Kuopio, Dept of Computer Science Pekka.Kilpelainen@cs.uku.

1 Querying XML Documents and Data CBU Summer School (2 ECTS) Prof. Pekka Kilpeläinen inen Univ of Kuopio, Dept of Computer Science Pekka.Kilpelainen@cs.uku.fi Introduction & Motivation order XML Internet XML appears everywhere How to query it? invoice CBU Summerschool '07 Querying XML: Introduction 2 Main Topic: Two XML Query Models Course Outline Region algebra for retrieval of structuded text "lightweight"» reduced language; for ad-hoc files; efficient free implementation XQuery for general querying/manipulation of XML "heavy"» comprehensive and complex language; for (data viewed as) XML only; production-use use implementations? CBU Summerschool '07 Querying XML: Introduction 3 Intro and Arrangements; Structured documents 1 Review of XML Basics 1.1 XML and XML docs; 1.2 Document grammars 1.3. XML DTDs; ; 1.4 XML Namespaces 1.5 XML Schema 2 Region Algebra and sgrep 3 W3C XQuery,, and XPath 2.0 (Apologies for potential dis-organization!) CBU Summerschool '07 Querying XML: Introduction 4 Arrangements Background: W3C Recommendations (XML, XQuery) Reports (Region Algebra, sgrep) Earlier courses; own research and experiments Some material (to be posted) at Plan: Lectures 12 h; hands-on exercises 8 h Structured Documents Document: a structured representation of information on some medium ( ( message) normally for a human reader» memos, manuals, articles, books, also application-to to-application messages» e.g., btw client and server in Web Services "prose-oriented oriented XML" vs "data-oriented XML" can be treated as a single unit» (a web page vs a web site) CBU Summerschool '07 Querying XML: Introduction 5 CBU Summerschool '07 Querying XML: Introduction 6 Presentation vs Structure Presentation informs the human reader about the meaning of text and the role of its parts Markup indicates the presentation or the meaning of different parts of text» originally hand-written annotations for the typesetter nowadays primarily codes embedded in digital documents; <Tags> CBU Summerschool '07 Querying XML: Introduction 7 Markup and Markup Language Procedural markup commands (start boldface, produce empty line, indent 5 mm,...) proprietary word processor formats, nroff, TeX,,... Descriptive or generic markup indicates conceptual structures using chosen names LaTeX: \begin{abstract}... \end{abstract} HTML: <TITLE>... </TITLE> Markup language a fixed set of markup notations (e.g. nroff, TeX,, HTML, SVG, ) CBU Summerschool '07 Querying XML: Introduction 8

2 Structure in Documents Hierarchy or nesting is ubiquitous Sections w. subsections etc (Also overlapping hierarchies!) Linear order essential in prose documents less important in documents representing data objects Hypertext and cross-references references XML: proper hierarchies, tree-like structures, with cross-references references via attribute values 1 Document Instances and Grammars Overview of fundamentals, and some details, of XML 1.1 XML and XML documents 1.2 Basics of document grammars 1.3 Basics of XML DTDs 1.4 XML Namespaces 1.5 XML Schema CBU Summerschool '07 Querying XML: Introduction 9 CBU Summerschool '07 Querying XML: Introduction XML and XML documents What is XML? XML - Extensible Markup Language, W3C Recommendation, February 1998 not an official standard, but a stable industry standard 2 nd Ed 2000, 3 rd Ed 2004, 4 th Ed 2006» editorial revisions, not new versions of XML 1.0 a simplified subset of SGML, Standard Generalized Markup Language, ISO 8879:1987 valid XML documents are also SGML documents Extensible Markup Language is not a markup language! does not fix a tag set nor its semantics (like markup languages like HTML do) XML documents have no inherent (processing or presentation) semantics even though many think that XML is semantic or self- describing; See next CBU Summerschool '07 Querying XML: Introduction 11 CBU Summerschool '07 Querying XML: Introduction 12 Semantics of XML Markup What is XML (2)? Meaning of this XML fragment? The application has to understand the tags But better off with the tags, though! CBU Summerschool '07 Querying XML: Introduction 13 XML is a way to use markup to represent information a metalanguage» supports definition of specific markup languages through XML DTDs (Document Type Definitions) or Schemas» E.g. XHTML a reformulation of HTML using XML Often XML XML + XML technology CBU Summerschool '07 Querying XML: Introduction 14 How does it look? Essential Features of XML <?xml version= encoding= iso iso ?> <invoice num= > <client clnum= > <name>pekka Kilpeläinen</ inen</name> < >kilpelai@cs.uku.fi</ </ > </client client> <item price= unit= EUR EUR > XML Handbook</ </item> <item price= unit= FIM FIM > XSLT Programmer s Ref</ </item> </invoice invoice> CBU Summerschool '07 Querying XML: Introduction 15 Overview of XML essentials many details skipped Learn to consult original sources (specifications, documentation etc) for details!» The XML specification is easy to browse First of all, XML is a textual or character-based way to represent data CBU Summerschool '07 Querying XML: Introduction 16

3 XML Document Characters External Aspects of Characters XML documents are made of ISO (32-bit) characters; ; in practice of their 16-bit Unicode subset (used, e.g., in Java) Unicode 2.0 defines almost 39,000 distinct characters Characters have three different aspects: their identification as numeric code points their representation by bytes their visual presentation Documents are stored/transmitted as a sequence of bytes (of 8 bits). An encoding determines how characters are represented by bytes. UTF-8 8 ( 7-bit( ASCII) is the XML default encoding encoding="koi8r" should be OK for Cyrillic texts» (I cannot comment on parser support) A font determines the visual presentation of characters CBU Summerschool '07 Querying XML: Introduction 17 CBU Summerschool '07 Querying XML: Introduction 18 XML Encoding of Structure 1 XML Encoding of Structure 2 XML is, essentially, a textual encoding scheme of labelled, ordered and attributed trees: internal nodes are elements labelled by type names leaves are text nodes labelled by string values, or empty element nodes the left-to to-right order of children of a node matters element nodes may carry attributes = (name, string-value) pairs This view is shared by many XML techniques (DOM, XPath,, XSLT, XQuery,,...) CBU Summerschool '07 Querying XML: Introduction 19 XML encoding of a tree corresponds to a pre-order walk start of an element node with type name A denoted by a start tag <A>, and its end denoted by end tag </A> possible attributes written within the start tag: <A attr 1 = value 1 attr n = value n >» Names attr 1,,attr n must be distinct text nodes written as their string value CBU Summerschool '07 Querying XML: Introduction 20 XML Encoding of Structure: Example XML: Logical Document Structure W Hello S E A=1 <S> <W> Hello </W><E <E A= 1 /> <W> world! </W> </S> W world! Elements indicated by matching (case-sensitive!) sensitive!) tags <ElementTypeName> </ElementTypeName ElementTypeName> can contain text and/or subelements can be empty: <elem-type></ type></elem-type> or <elem-type/> (e.g. <br/> in XHTML) unique root element > document a single tree CBU Summerschool '07 Querying XML: Introduction 21 CBU Summerschool '07 Querying XML: Introduction 22 Logical document structure (2) Attributes name-value pairs attached to elements in start-tag tag after the element type name <div class="preface" date='990126'> forms "..." and '...' are interchangeable Also: <div class="preface" date='990126'> comments outside other markup --> <?note this would be passed to the application as a processing instruction named note?> CBU Summerschool '07 Querying XML: Introduction 23 CDATA Sections CDATA Sections to include XML markup characters as textual content <![CDATA[ Here we can easily include markup characters and, for example, code fragments: <example>if (Count < 5 && Count > 0) </example> ]]> CBU Summerschool '07 Querying XML: Introduction 24

4 Two levels of correctness (1) XML docs and valid XML docs Well-formed documents roughly: follows the syntax of XML, markup correct (elements properly nested, tag names match, attributes of an element have unique names,...) violation is a fatal error Valid documents (in addition to being well-formed) obey an associated grammar (DTD/Schema) CBU Summerschool '07 Querying XML: Introduction 25 DTD-valid documents Schema-valid documents XML documents = well-formed XML documents CBU Summerschool '07 Querying XML: Introduction 26 An XML Processor (Parser) 1.2 Basics of document grammars Reads XML documents and reports their contents to an application relieves the application from details of markup XML Recommendation specifies: recognition of characters as markup or data; what information to pass to applications; how to check the correctness of documents; validation based on comparing document against its grammar Next: Basics of document grammars CBU Summerschool '07 Querying XML: Introduction 27 DTDs are variations of context-free grammars (CFGs), which are widely used to syntax specification (programming languages, XML, ) and to parser/compiler generation (e.g. YACC/GNU Bison) No knowledge of them is necessary, but connections with CFGs may be informative for those that know about them CBU Summerschool '07 Querying XML: Introduction 28 DTD/CFG Correspondence Example: Three Authors of a Ref DTD XML document element type element type declaration #PCDATA CFG parse/syntax syntax tree nonterminal production terminal Author Author Aho Hopcroft Ref Author Title PublData... Ullman The Design and Analysis... Ref > Author Title PublData P, Author Author Author Title PublData L(Author Title PublData) CBU Summerschool '07 Querying XML: Introduction 29 CBU Summerschool '07 Querying XML: Introduction 30 Extended Productions 1.3 Basics of XML DTDs Notice the regular expressions in productions to describe (potentially infinite) sequences That is, we are using extended CFGs content models (of a DTD) correspond to regular expressions (in an ECFG production) > number of element s s children generally unlimited A Document Type Declaration provides a grammar (document( type definition, DTD) ) for a class of documents [Defined in XML Rec] Syntax (in the prolog of a document instance): <!DOCTYPE rootelemtype SYSTEM "ex.dtd" "external subset" in file ex.dtd --> [ <! - an optional "internal subset" --> ] > DTD = union of the external and internal subset internal has preference for attribute and entity decls CBU Summerschool '07 Querying XML: Introduction 31 CBU Summerschool '07 Querying XML: Introduction 32

5 Markup Declarations DTD consists of markup declarations element type declarations» productions of ECFGs attribute-list declarations» for declared element types entity declarations» for physical structures notation declarations logical structures CBU Summerschool '07 Querying XML: Introduction 33 How do Declarations Look Like? <!ELEMENT invoice (client, item+)> <!ATTLIST invoice num NMTOKEN #REQUIRED> <!ELEMENT client (name, ?)> <!ATTLIST client num NMTOKEN #REQUIRED> <!ELEMENT name (#PCDATA)> <!ELEMENT (#PCDATA)> <!ELEMENT item (#PCDATA)> <!ATTLIST item price unit NMTOKEN #REQUIRED (FIM EUR) EUR > CBU Summerschool '07 Querying XML: Introduction 34 Element Type Declarations General form: <!ELEMENT elementtypename (E)> where E is a content model regular expression of element names Content model operators: E F : choice E, F: concatenation E? : optional E : zero or more E+ : one or more (E) : grouping Must group: (A,B) C or A,(B C), but A,B C forbidden CBU Summerschool '07 Querying XML: Introduction 35 Attribute-List Declarations Can declare attributes for elements: Name, data type and possible default value Example: <!ATTLIST FIG id ID #IMPLIED descr CDATA #REQUIRED class (a b c) "a Semantics mainly up to the application processor checks that ID attributes are unique and that targets of IDREF attributes exist CBU Summerschool '07 Querying XML: Introduction 36 Mixed, Empty and Arbitrary Content Mixed content: <!ELEMENT P (#PCDATA I IMG)> may contain text and elements Empty content: <!ELEMENT IMG EMPTY> Unrestricted content: ANY (= (#PCDATA choice-of of-all-declared-element-types) ) Entities (1) Named storage units of XML documents Multiple uses: character entities:» < < and < all expand to < (treated as data, not as start-of of-markup)» other predefined entities: & &gt& gt; ; &apos& apos; ; &quote; expand to &, >, ' and " general entities are shorthand notations: <!ENTITY UKU "University of Kuopio CBU Summerschool '07 Querying XML: Introduction 37 CBU Summerschool '07 Querying XML: Introduction 38 Entities (2) physical storage units comprising a document parsed entities <!ENTITY chap1 SYSTEM " document entity is the starting point of processing entities and elements must nest properly: <!DOCTYPE doc [ <!ENTITY chap1 ( as above ) > ]> <doc> &chap1; </doc> <sec num="1 </sec> <sec num="2 </sec> CBU Summerschool '07 Querying XML: Introduction 39 Unparsed Entities and Parameter Entities Unparsed entities allow XML documents refer to external binary objects like graphics files XML processor handles only text I've rarely used these Parameter entities are used in DTDs useful for modularizing declarations We skip these CBU Summerschool '07 Querying XML: Introduction 40

1.4 XML Namespaces Documents often comprise parts processed by different applications (and/or defined by different grammars) for example, in XSLT scripts: <xsl:template match="doc/title <H1> HTML

6 1.4 XML Namespaces Documents often comprise parts processed by different applications (and/or defined by different grammars) for example, in XSLT scripts: <xsl:template match="doc/title <H1> HTML <xsl:apply-templates /> elements </H1> XSLT </xsl:template xsl:template> elements/ instructions How to manage multiple sets of names? CBU Summerschool '07 Querying XML: Introduction 41 XML Namespaces (2/5) Solution: By introducing (arbitrary) local name prefixes, and binding them to (fixed) globally unique URIs For example, the local prefix xsl: conventionally used in XSLT scripts CBU Summerschool '07 Querying XML: Introduction 42 XML Namespaces briefly (3/5) XML Namespaces (4/5) Namespace identified by a URI (through the associated local prexif) e.g. for XSLT conventional but not required to use URLs the identifier has to be unique, but no need to be an address Association inherited to sub-elements see the next example (of an XSLT script) <xsl:stylesheet version="1.0" xmlns:xsl=" xmlns=" :// XHTML is the default namespace --> <xsl:template match="doc/title <H1> <xsl:apply-templates /> </H1> </xsl:template xsl:template> </xsl:stylesheet xsl:stylesheet> CBU Summerschool '07 Querying XML: Introduction 43 CBU Summerschool '07 Querying XML: Introduction 44 XML Namespaces briefly (5/5) 1.5 XML Schemas A quick look at XML Schema W3C Recommendation, 1 st Mechanism built on top of basic XML overloads attribute syntax (xmlns( xmlns:) ) to introduce namespaces does not affect validation» namespace attributes have to be declared for DTD- validity» all element type names have to be declared (with their initial prefixes!) > Other schema languages (XML Schema, Relax NG) better for validating documents with Namespaces st Ed. May, 2001; 2 nd Ed. Oct, 2004:» XML Schema Part 0: Primer (readable non- normative introduction; Recommended)» XML Schema Part 1: Structures» XML Schema Part 2: Datatypes W3C Draft (didn't lead anywhere?):» Formal Description,, 9/2001 CBU Summerschool '07 Querying XML: Introduction 45 CBU Summerschool '07 Querying XML: Introduction 46 Advantages of XML Schema (1) XSDL built-in in types (Part 2, Chap. 3) XML syntax easier to manipulate by programs (than DTDs) Compatibility with namespaces can validate against declarations from multiple sources Content datatypes 44 built-in in datatypes (including primitive Java datatypes, datatypes of SQL, and XML attribute types) mechanisms to derive user-defined datatypes used as types of XQuery CBU Summerschool '07 Querying XML: Introduction 47 CDATA NB: all simple values in documents strings : XML attribute types CBU Summerschool '07 Querying XML: Introduction documents 48

7 Advantages of XML Schema (2) Advantages of XML Schema (3) Element names and content types independent; Compare with For example, could define titles» of people as Mr. Mr. / Mrs. Mrs. / Ms. Ms.,, and» of chapters as strings > extend the power of CFGs/DTDs» where non-terminal / tag-name alone determines its allowed content (Is this relevant in practice?) Ability to specify uniqueness and keys within selected parts of the document for example, that titles of chapters should be unique; or key attributes of relations uses XPath Support for schema documentation element annotation with sub-elements documentation (for human readers) and appinfo (for applications) Only these contain text (#PCDATA) CBU Summerschool '07 Querying XML: Introduction 49 CBU Summerschool '07 Querying XML: Introduction 50 Disadvantages of XML Schema XSDL through Example Complexity (esp. Rec Part 1!) vs. added power > a long learning curve > slow adoption by users Immaturity of implementations (?) W3C web site mentions ~ 60 tools/processors Apache Xerces claims full XSDL support Some features difficult to implement efficiently Alternative schema languages have been suggested, too CBU Summerschool '07 Querying XML: Introduction 51 Next: walk-through of an XML schema example from Chapter 2 of the XML Schema Primer Consider modelling purchase orders like below: <purchaseorder orderdate=" <shipto country="us <name>alice Smith</name name> <street>123 Maple Street</ </street> <city>mill Valley</city> <state>ca</ </state> <zip>90952</zip </shipto shipto> CBU Summerschool '07 Querying XML: Introduction 52 purchaseorder instance continues End of the example instance <billto country="us <name>robert Smith</name name> <street>8 Oak Avenue</street street> <city>old Town</city> <state>pa</ </state> <zip>95819</zip > </billto billto> <comment>hurry,, my lawn is wild!</!</comment> <items><item partnum="872-aa <productname>lawnmower</ </productname> <quantity>1</quantity <USPrice>148.95</USPrice <comment>only if electric</ </comment> </item item> CBU Summerschool '07 Querying XML: Introduction 53 <item partnum="926-aa <productname>baby Phone</ </productname> <quantity>1</quantity <USPrice>39.98</USPrice <shipdate> </ 21</shipDate> </item item> </items items> </purchaseorder purchaseorder> Next: A schema for such purchase orders CBU Summerschool '07 Querying XML: Introduction 54 The Purchase Order Schema (1/5) <xs:schema xmlns:xs=" :// name="purchaseorder" type="pordtype name="comment" type="xs:string <xs:complextype name="pordtype <xs:sequence> name="shipto" type="usaddr name="billto" type="usaddr ref="comment" " minoccurs="0 name="items" type="items </xs:sequence xs:sequence> <xs:attribute name="orddate" type="xs:date </xs:complextype xs:complextype> CBU Summerschool '07 Querying XML: Introduction 55 The Purchase Order Schema (2/5) <xs:complextype name="usaddr <xs:sequence> name="name" type="xs:string name="street" type="xs:string name="city" type="xs:string name="state" type="xs:string name="zip" type="xs:decimal </xs:sequence xs:sequence> <xs:attribute name="country" type="xs:nmtoken" fixed="us </xs:complextype xs:complextype> CBU Summerschool '07 Querying XML: Introduction 56

8 The Purchase Order Schema (3/5) <xs:complextype name="items <xs:sequence> name="item" minoccurs="0" maxoccurs="unbounded <xs:complextype> anonymous type for item <xs:sequence> name="productname" type="xs:string name="quantity <xs:simpletype> <xs:restriction anon. type base="xs:positiveinteger for quantity <xs:maxexclusive value="100 </xs:restriction xs:restriction> </xs:simpletype xs:simpletype> </xs:element xs:element> CBU Summerschool '07 Querying XML: Introduction 57 The Purchase Order Schema (4/5) name="usprice" type="xs:decimal ref="comment" minoccurs="0 name="shipdate" type="xs:date" " minoccurs="0 </xs:sequence xs:sequence> <xs:attribute name="partnum" type="sku" use="required </xs:complextype xs:complextype> </xs:element xs:element> > item --> </xs:sequence xs:sequence> </xs:complextype xs:complextype> > Items --> CBU Summerschool '07 Querying XML: Introduction 58 The Purchase Order Schema (5/5) Type for Stock Keeping Units, (codes for identifying products): --> <xs:simpletype name="sku <xs:restriction base="xs:string defined by a regular expr: --> <xs:pattern value="\d{3} d{3}-[a-z]{2}" /> 3 digits, hyphen,, 2 letters --> </xs:restriction xs:restriction> </xs:simpletype xs:simpletype> </xs:schema xs:schema> CBU Summerschool '07 Querying XML: Introduction 59 XML Schema: Summary XSDL: an XML-based grammar formalism W3C Recommendation; Alternative to DTDs» support for namespaces» richer content and attribute datatypes Well accepted(?) in XML industry e.g., to describe messages btw clients and servers in Web services; ; (See, e.g., Web Services Description Language, Vers.. 2.0, W3C Draft 3/07) for typing of XQuery CBU Summerschool '07 Querying XML: Introduction 60

SDPL : XML Basics 2. SDPL : XML Basics 1. SDPL : XML Basics 4. SDPL : XML Basics 3. SDPL : XML Basics 5

SDPL : XML Basics 2. SDPL : XML Basics 1. SDPL : XML Basics 4. SDPL : XML Basics 3. SDPL : XML Basics 5 2 Basics of XML and XML documents 2.1 XML and XML documents Survivor's Guide to XML, or XML for Computer Scientists / Dummies 2.1 XML and XML documents 2.2 Basics of XML DTDs 2.3 XML Namespaces XML 1.0