Chapter 5 Mapping XML to Databases

Transcription

1 Chapter 5 Mapping XML to Databases Three Layer Architecture Text-based Storage Model-based Storage Structure-based Storage Hybrid Storage

2 Three Layer Architecture for Database Processing Conceptual Layer Logical Layer Physical Layer Abstract database schema e.g. ER schema Abstract Instances e.g. entities Concrete database schema e.g. table schema Concrete Instances e.g. tables and tuples Storage structures and access paths e.g. B-tree index Data sets in database pages Conceptual Layer: ERM, UML,... Logical Layer: relational model, NF2 model, OR model,... Physical Layer: internal storage, indexing, updates, transaction processing, query optimization 5-2

3 Three Layer Architecture for Document Processing Conceptual Layer Logical Layer Physical Layer Style sheet Document structure File Document Access structure e.g. full-text index Conceptual Layer: editors basic agreement on the document structure (intro before conclusions etc.) Logical Layer: document model, document instances Physical Layer: internal storage, document indexing, query processing 5-3

4 Three Layer Architecture for XML Processing Conceptual Layer Logical Layer Physical Layer <..> </..> <..>? </..> <..> </..> Document structure e.g. XML schema XML documents, XML elements Storage structure Access structure Conceptual Layer: XML schema, DTD Logical Layer: XML documents, XML elements, etc. Physical Layer - Storage: Single file inappropriate for large document collections Hot to store large XML documents? How to support queries efficiently? 5-4

5 Types of XML Documents Structured regular example: product catalogue, orders, invoices data-centric applications Unstructured irregular example: articles, books, s, Web pages document-centric applications <order> <customer>meyer</customer> <position> <isbn> </isbn> <number>2</number> <price currency="euro">30.00</price> </position> </order> <content> XML builds on the principles of two existing languages, <em>html</em> and <em>sgml</em> to create a simple mechanism... The generalized markup concept... </content> Semi-structured structured and unstructured parts example: publications, Amazon, MS Press (example chapters) <book> <author>neil Bradley</author> <title>xml companion</title> <isbn> </isbn> <content> XML builds on the principles of two existing languages, <em>html</em> and... </content> </book> 5-5

6 Three Layer Architecture and XML Document Types document-centric semi-structured data-centric Conceptual Layer Modeling structure and content Tree-based editors Modeling structure and content XML Modeling structure ER, UML Document model Data and document model Data model Logical Layer Queries/Updates on structure and content XML, SGML Queries/Updates on structure and content XML, OEM Queries/Updates on content XML, RDM, OODM XPath/XQuery, DOM, IR queries XPath/XQuery, XIRQL XQuery, SQL, OQL Physical Layer Structure on instance level Files and full-text indexes, structure indexes Structure on schema and instance level Semi-structured databases and full-text indexes, path indexes Structure on schema level (Object-)Relational databases and B-tree / hash indexes 5-6

7 Requirements on the Physical Layer Order preserving and lossless storage of XML documents Efficient access to XML documents and selected document parts Short response times for Queries Updates Physical design of the storage layer Indexing Transaction processing Support for XPath/XQuery Support for XML processors such as DOM and SAX 5-7

8 Storage Strategies for XML Documents 1. Text-based Store XML document as string 2. Model-based Store XML graph structure and DOM informations 3. Structure-based Derive database schema from XML structure Custom mapping of XML structure to database schema 5-8

9 Text-based Storage Entire XML document stored as File in the file system CLOB (Character Large Object) in a database system Id Content 1 <book genre="autobiography"> <title>the Autobiography of Benjamin Franklin</title> <author> <first-name>benjamin</first-name> <last-name>franklin</last-name> </author> <price currency="usd">8.99</price> </book> Operations (read/write) on entire XML document are efficient No explicit support for more fine-granule access; use of indexes required! Full-text index Path indexes 5-9

10 Summary Text-based Storage Schema Not required Document reconstruction Not needed since documents are stored entirely as string Queries Information Retrieval queries XML queries possible Further Specifics Full-text functionality Efficiency String must be parsed at access time time-consuming No concurrency for read and write operations no parallel executions Application area Document-centric XML applications Storing semi-structured XML documents 5-10

11 Model-based Storage Generic storage of XML document tree XML elements, XML attributes, etc. are nodes of tree Store XML elements/attributes in corresponding database tables Element ID Name Value Parent Element Position Attribute ID Name Value Parent Element Extensions for adequate storage of different data types possible 5-11

12 Model-based Storage: EDGE, BINARY XML title store auction 5.20 XML documents price Stars title store auction BINARY variant: horizontal partitioning of EDGE along the values of the column label price CREATE TABLE edge ( id INTEGER NOT NULL, doc INTEGER NOT NULL, -- document id parent INTEGER, label VARCHAR(30) NOT NULL, content VARCHAR(100), nodetype VARCHAR(20) NOT NULL, pos INTEGER,-- element position PRIMARY KEY (id, doc), UNIQUE (parent, pos), CHECK (nodetype<>ˈattributeˈ and pos is null) ); id doc parent label content nodetype pos 1 1 null store null element auction null element title XML element price 5.20 element null store null element auction null element title Stars element price element

13 Model-based Storage: Queries XML queries XQuery SQL DB XML (XPath, XQuery) queries internally transformed to SQL queries XML query result derived from the SQL result Labelling of the resulting tuples Result must be an XML value Example: Get the description of all auctions that have a price less than 10 SELECT a.content, b.content FROM Edge a, Edge b WHERE (a.label = 'price') AND (a.content < 10.00) AND (b.label = 'title') AND (b.parent = a.parent) AND (a.doc = b.doc) id doc parent label content nodetype pos 1 1 null store null element auction null element title XML element price 5.20 element null store null element auction null element title Stars element price element

14 Model-based Storage: DOM (1) DOMImplementation Node NodeList NamedNodeMap Information of the Document Object Model stored in a database Attr CharacterData Comment Text CDataSection Document DocumentFragment DocumentType Element Entity EntityReference Notation ProcessingInstruction 5-14

15 Model-based Storage: DOM (2) Node type: ELEMENT 1 book Node type: ATTRIBUTE 2 title 3 price NodeID NodeType DocID ParentNode 1 ELEMENT 1 NULL 2 ELEMENT ATTRIBUTE TEXT 1 2 NodeID TagName NodeID Content The Autobiography of Benjamin Franklin book 2 title 4 The Auto... Node type: TEXT NodeID PreviousSibling NextSibling FirstChild 1 NULL NULL 2 2 NULL NULL 4 3 NULL NULL NULL NodeID ElementID AttributeName AttributeValue 3 1 price

16 Model-based Storage: DOM Methods XML queries DOM SQL DB XML queries (DOM method calls) mapped to corresponding SQL queries Result of the method call derived from the result of the SQL query 5-16

17 Summary Model-based Storage Schema Not required Document reconstruction Possible but expensive Queries XML queries possible Adapted database queries Queries over many elements/attributes are expensive Efficiency Context-based navigation is efficient Document reconstruction expensive Path expression evaluation inefficient Application area Data-centric Document-centric Semi-structured 5-17

18 Structure-based Storage Idea: Shred XML documents and map the parts to corresponding database structures If available derive database schema from a DTD or XML schema Tolerate loss of native XML access Problem Automatic generation of database schema Exploiting structure information in queries Example: Basic mapping from a DTD to a database schema Transform DTD in tree representation Nodes represent element types, attributes, etc., i.e., the type layer! Edges represent relationships between element types as well as certain restrictions 5-18

19 Mapping DTD to Databases Rules for mapping element types: XML element type table column Sequence of element types table column Alternative of element types table column Element type with quantifier? table column allowing nulls Element type with quantifier +,* set/list-valued table column Nested element type tuple-valued table column Rules for mapping attributes: XML attribute table column IMPLIED NULLABLE REQUIRED NOT NULLABLE Default value DEFAULT Constraint 5-19

20 Mapping XML onto Relational Databases <hotel> <name>eden</name> <address> <zip>8008</zip> <city>zurich</city> <street>bellerive</street> <no>12</no> </address> <phone> </phone> <phone> </phone> </hotel> Schema typically needed Queries use SQL RDBMS data types Hotel Variant 1 Id Name Zip City Street No 1 Eden 8008 Zurich Bellerive 12 Phone Id Text Hotel Hotel Variant 2 Id Name Address 1 Eden 101 Address Id Zip City Street No Zurich Bellerive 12 Mapping collection types to separate tables Mapping tuple types implicitly (Variant1; tuple structure lost) or separately (Variant 2) 5-20

21 Mapping XML onto Object-Relational Databases <hotel> <name>eden</name> <address> <zip>8008</zip> <city>zurich</city> <street>bellerive</street> <no>12</no> </address> <phone> </phone> <phone> </phone> </hotel> Hotel ID Name <Address> {Phone} Zip City Street No 1 Eden 8008 Zurich Bellerive 12 { , } Schema typically required Queries use SQL Straight-forward mapping of tuple and collection types to corresponding database data types 5-21

22 Mapping XML onto Object-Relational Databases (2) XML-DTD <!ELEMENT book (title, author+, edition?, publisher)> <!ATTLIST book isbn CDATA #REQUIRED> <!ELEMENT title (#PCDATA)> <!ELEMENT author (first, last)> <!ELEMENT first (#PCDATA)> <!ELEMENT last (#PCDATA)> <!ELEMENT edition (#PCDATA)> <!ELEMENT publisher (#PCDATA)> Informix-Syntax: CREATE TABLE book ( isbn VARCHAR(20) NOT NULL, title VARCHAR(100) NOT NULL, author LIST(ROW( first VARCHAR(30), last VARCHAR(50)) NOT NULL), edition INTEGER, publisher VARCHAR(100) NOT NULL, ); ISBN Title {<Author>} Edition Publisher First Last XML & Databases Meike Klettke 1 dpunkt Holger Meyer Problem: Map to an SQL schema that allows only XML documents which are valid w.r.t. the given DTD 5-22

23 Mapping XML Alternatives XML DTD supports alternatives Example: <!ELEMENT author (name ((first-name, last-name)))> Mapping variants: 1. Each alternative is mapped to a separate column of a common table Parse tables with many null values 2. Each alternative is mapped to a separate table Queries require union over several tables 3. Map to a single table column of type XML <author> <first-name>benjamin</first-name> <last-name>franklin</last-name> </author> <author> <first-name>herman</first-name> <last-name>melville</last-name> </author> <author> <name>plato</name> </author> Variant 1 Variant 2 Name FirstName LastName NULL Benjamin Franklin NULL Herman Melville Plato NULL NULL Name FirstName LastName Plato Benjamin Franklin Herman Melville 5-23

24 Mapping Recursive XML Elements XML DTD supports recursive XML elements In database no endless recursion on instance level possible Example: <!ELEMENT book (front, body, references)> <!ELEMENT references (book+)> <book>... <references> <book>... <references>... </references> </book> </references> </book> book <references> book references ID book references Solution: Map recursive XML elements to corresponding tables for each element type and use primary/foreign keys and reference types, respectively, to connect these tables 5-24

25 Mapping XML Element Ordering Ordering is relevant in XML; must therefore preserved Simple mapping rule: Add additional table column to hold the position of each element Example: <book> <chapter>introduction</chapter> <chapter>basic</chapter> <chapter>concept</chapter> <chapter>example</chapter> <chapter>conclusions</chapter> </book> Chapter Position Introduction 1 Basics 2 Concept 3 Example 4 Conclusions

26 Mapping ANY, Mixed Content No explicit representation available for content model ANY Appropriate storage structure can only be selected on the instance level Use text-based storage variants for mixed content 1. One column for entire document MixedContent <description> Reach us at FGCZ: <tram>10 min from SBB Zurich</tram> <car>20 min from SBB Zurich</car> Check </description> 2. Separate columns for reoccurring element types Position #PCDATA Tram Car 1 Reach us at FGCZ: NULL NULL 2 NULL 10 min from SBB Zurich NULL 3 NULL NULL 20 min from SBB Zurich 4 Check NULL NULL 5-26

27 STORED: Semi-structured TO Relational Data Mapping with STORED Basic idea: apply data mining on XML documents to get a good mapping onto database tables Approach: Input XML documents (at least a representative probe of the given XML document collection) Query workload Restrictions: hard disk space, number of database tables, etc. Output Relational database schema STORED queries: rules for XML documents onto database tables Algorithm Determine the parts of the XML tree parts that have the highest support in the document collection and queries These tree parts are explicitly mapped onto own tables Irregular data mapped onto common table Overflow which is stored using the EDGE approach Neither DTD nor XML schema required 5-27

28 Mapping with STORED: Example Tree parts with high support store auction XML documents store auction STORED query FROM '//auction' { 'title' : v1 'price' : v 2 } STORE auction(v 1,v 2 ) title price title price unusual Side Table auction doc title price XML 5.20 Stars XYZ 1 XML Stars Overflow Table doc ordinal name datatype value 2 3 unusual string XYZ 5-28

29 Structure-based Storage: Querying XML queries XQuery SQL DB XML (XQuery) queries mapped to corresponding SQL queries Mapping XML query to database query must be generated automatically Process of mapping XML documents onto databases must be logged Database query results must be mapped XML query results 1-29

30 Structure-based Storage with Automatic Mapping Schema Required for DTD/XML schema mapping Not needed in case of STORED Document reconstruction Logging mapping process including inverse mapping required not always possible Queries Database queries XML queries No full-text operations Usually no native access Further Specifics Big schema weakly filled database tables (many null values) No flexible data types, no explicit storage support for alternatives Efficiency High due to full exploitation of the DBMS Application area Data-centric XML applications 5-30

31 Structure-based Storage with Custom Mapping User determines storage structures using custom mapping rules XML document structure and database schema can be designed independently Example: Price <Hotel url=" <Name>Eden</Name> <Address> <Zip>8008</Zip> <City>Zurich</City> <Street>Bellerive</Street> <No>12</No> </Address> <Price>198</Price> </Hotel> Hotel CHF Link Eden <ClassMap> <ElementType Name="Hotel"/> <ToClassTable> <Table Name="Price"> </ToClassTable> <PropertyMap> <Element Name="Name"/> <ToColumn Name="Hotel" /> </PropertyMap> <PropertyMap> <Element Name="Price"/> <ToColumn Name="CHF" /> </PropertyMap> <PropertyMap> <Attribute Name="url"/> <ToColumn Name="Link" /> </PropertyMap> </ClassMap> XML queries possible if inverse mapping rules can be derived from the logged mapping rules Inappropriate if full document reconstruction is required! 5-31

32 Summary Structure-based Storage with Custom Mapping Schema Required Document reconstruction Requires inverse mappings Usually not possible Efficiency High due to DBMS Application area Data-centric applications Queries Database queries XML queries possible in restricted cases 5-32

33 Hybrid Storage Text-based Storage Model-based Storage Structure-based Storage Full-Text Index Full-Text Index and XML Index Standard Mapping Custom Mapping Shredding Degree document-centric semi-structured data-centric Detecting data-centric and document-centric document parts Use different storage approaches for the different document parts 1-33

34 Hybrid Storage XML documents with data-centric and document-centric parts are difficult to map Storage strategies for data-centric parts inappropriate for document-centric parts and vice versa None approach satisfies all requirements Hybrid storage: use different storage models for different document parts Combining text-based and structure-based storage Combining structure-based storage and database type XML Structured parts mapped to object-relational structures Unstructured parts mapped to objects of type XML Problem: Detecting and separating structured from irregular document parts Basic idea of a hybrid approach to create a database schema Build DTD or XML schema graph For each graph node, determine a mass representing the significance of the node Set shredding degree to determine granularity of resulting database schema Derive table columns of type XML Map structured document parts onto corresponding object-relational structures 5-34

35 Conclusions Mapping Strategies for Storing XML Documents Text-based Store entire XML document as string Plus full-text and/or path indexing Model-based Store the XML graph model Structure-based Analyze structure of XML documents to derive a database schema from the XML structure Hybrid Combination of strategies above No one-fits-all solution! Appropriate storage solution mainly depends on the application area! 5-35