CS490W XML data and Retrieval XML and Retrieval: Outline

Transcription

1 CS490W XML data and Retrieval Luo Si Department of Computer Science Purdue University XML and Retrieval: Outline Outline: Semi-Structure Data XML, Examples, Application XML Search XQuery XIRQL Text-Based XML Retrieval Vector-space model INEX

2 Semi-Structured Data XML has been used as the standard representation of Semi- Structured Data extensible Markup Language is a W3C-recommended general-purpose markup language that supports a wide variety of applications. A framework for defining markup languages Open vocabulary for tags Each set of XML corresponds to different applications facilitate the sharing of data across different information systems, particularly systems connected via the Internet Examples: RSS, XHTML, MathML Semi-Structured Data Structure of XML XML data is organized by documents like unstructured data There are structures (nodes/tags) within the documents Each XML document is an ordered, labeled tree Element Nodes are labeled with Node name (e.g., chapter) Node attributes and the values (e.g., size=1000; time=01/01/2007) May have child nodes or data Data exist (e.g., text strings) within leaf nodes

3 XML Example <book id= ML_Tom > <title>machine Learning</title> <author> <firstname>tom</firstname> <surname>mitchell</surname> </author>... <p>machine Learning Applications...</p>... </book> Elements, Attributes/Values, Data(Text String) XML Example <book id= ML_Tom > <title>machine Learning</title> <author> <firstname>tom</firstname> <surname>michael</surname> </author>... <p>machine Learning Applications...</p>... </book> Elements, Attributes/Values, Data(Text String) title firstname author surname title book chapter para para chapter para

4 Elements Elements are defined by markup tags Elements: <TagName attr_a= value >text</tagname> ID of the element is TagName Attribute: attr_a; Values= value Data/text: text End tag </TagName> XML, HTML, SGML 1986: SGML ISO Nov 1995: HTML 2.0 Nov 1996: Simplified and stripped down SGML draft (dubbed XML) Jan 1997: HTML 3.2 Aug 1997: XML working draft Dec 1997: XML 1.0 proposed recommendation Jan 1998: XML Feb 1999: XHTML

5 XML and HTML Both of them are derivations of SGML HTML is a markup language mainly for display in browsers XML is a framework for markup languages HTML defines display XML defines the data structure, the display factor is separated from the content HTML can be formalized as XML (XHTML) Why XML? Unlike relational database, XML data does not require relational schemata, etc., because the data itself contains this information. Unlike widely used Web format, HTML, which only ensures the correct presentation of the formatted data, XML also guarantees total usability of data.

6 XML Applications CML chemical markup language: WML wireless markup language ThML theological markup language XML Applications CML chemical markup language: CML (Chemical Markup Language) is a new approach to managing molecular information using tools such as XML and Java. It was the first domain specific implementation based strictly on XML, <molecule convention="mdlmol" id="baclofen" title="baclofen">

7 XML Applications WML wireless markup language Wireless Markup Language, is a content format for devices that implement the Wireless Application Protocol (WAP) specification, such as mobile phones. <?xml version="1.0"?> <!DOCTYPE wml PUBLIC "-//PHONE.COM//DTD WML 1.1//EN" " > <wml> <card id="main" title="first Card"> <p mode="wrap">this is a sample WML page.</p> </card> </wml> XML Applications ThML theological markup language <ThML> <ThML.body> <div1> <div2 title="genesis" id="gen"> <div3 title="chapter 1"> <p> <scripture/> In the beginning God created the heaven and the earth. <scripture/> And the earth was without form, and void; and darkness was upon the face of the deep. And the Spirit of God moved upon the face of the waters. </p> </div3> </div2> </div1> </ThML.body> </ThML>

8 XML Files Schema/DTD: syntax definition of XML Language; Document Type Definition (DTD file) XML provides an application independent way of sharing data. With a DTD, independent groups of people can agree to use a common DTD for interchanging data. However, this is often NOT the case <?xml version="1.0"?> <!DOCTYPE note [ <!ELEMENT note (to,from,heading,body)> <!ELEMENT to (#PCDATA)> DTD Example <!ELEMENT from (#PCDATA)> <!ELEMENT heading (#PCDATA)> <!ELEMENT body (#PCDATA)> ]> XML Files <?xml version="1.0"?> <!DOCTYPE note [ <!ELEMENT note (to,from,heading,body)> <!ELEMENT to (#PCDATA)> <!ELEMENT from (#PCDATA)> <!ELEMENT heading (#PCDATA)> <!ELEMENT body (#PCDATA)> ]> <note> <to>tove</to> <from>jani</from> <heading>reminder</heading> <body>don't forget me this weekend!</body> </note> DTD Example XML Document

9 XML Files XML Schema: Recommended by the W3C as the successor of DTDs, more informally referred to by the initialism for XML Schema instances, XSD (XML Schema Definition). XSDs are far more powerful than DTDs in describing XML languages. <xs:schema xmlns:xs=" <xs:element name="country" type="country"/> <xs:complextype name="country"> <xs:sequence> <xs:element name="name" type="xs:string"/> <xs:element name="population" type="xs:decimal"/> </xs:sequence> </xs:complextype> </xs:schema> XML Search Most XML Search protocols use a database-based approach Non-text data match Exact keyword (text) match Evaluate XML path expression No concept of relevant

10 XML Search Traditional XML Search from Database-based approach XQuery Search multiple types of data: value-based (e.g., price of a book); ids (ISBN of book); keyword match (text) XML text search from information retrieval approach XIRQL Vector-space based Search text data: estimate relevance of xml elements with respect of query Query may contain path expressions XML Search XQuery SQL for XML Used for text-rich documents; data-oriented documents (non-text); mixed documents Consider: path expression (XPath); XML Schema datatypes It is still a working draft; details are being improved

11 XML Search XQuery considers some principal forms Path expression Conditional expressions Datatype expressions List expression etc Programming Language: Flowers (FLWOR) expression Principle forms can be evaluated with respect to context Principal Forms Path Query /book//title contains Information Retrieval title of the book contains keywords Information Retrieval Conditional expressions $h/title, IF = "Journal" THEN. if the type of an article is journal

12 Flowers (FLWR) Programming Language: Flowers (FLWR) expression The programming language XQuery defines FLWOR or FLWR (often pronounced as 'flower') as expression that supports iteration and binding of variables to intermediate results. For and let create a sequence of tuples where filters the tuples on a boolean expression order by sorts the tuples, using any comparable data return gets evaluated once for every tuple Flowers (FLWR) for $d in document("depts.xml")//deptno let $e := document("emps.xml")//employee[deptno = $d] where count($e) >= 10 order by avg($e/salary) descending return <big-dept> { $d, <headcount>{count($e)}</headcount>, <avgsal>{avg($e/salary)}</avgsal> } </big-dept>

13 XML Search XQuery considers some principal forms and combine them with Flowers (FLWR) It is quite similar to SQL for relational database However, it does not have the concept of relevance, which is important for both text data (text-based information retrieval) and non-text data (fuzzy search). Find a book about information retrieval Find a book which is about $30. XML IR Challenges 1: Term Statistics There are multiple types of elements: books/titles/abstracts; how to construct the corpus-statistics (idf) for different elements? How do we handle the term frequency information? Example: /book//title information retrieval do we consider the book abstract? Hierarchical smoothing

14 XML IR Challenges 2: Schemas Ideal Case There is a universal schema User can associate data type with the universal schema without ambiguity Too ideal to be true Real Word There are many schemas; different spellings; different concepts; different granularities; (e.g., auth & authors ; abstract & description ; abstract & keywords ) XML IR Challenges 3: User interface How to guide user to find relevant elements Granularity control: Book->Abstract->Full Text What type of querying language Natural language query (IR approach): most usable With structure information: more powerful but less usable How to do query expansion How to automatically add structure information e.g., find a book written by J. K. Rowling, -> find a book written by /../author (J. K. Rowling, ) open research problem

15 XIRQL Prof. Norbert Furth University of Dortmund: Open source XML search engine XIRQL: a query language for information retrieval in XML documents Structured Document Retrieval Principle Users may not know the schema Allow users to search even if they do not know the schema of the data Units Only atomic units can be returned traditional IR treats documents as atomic units; XML treat tree-like view of documents. XIRQL only indexes and returns atom-units Atom-units can be leaf nodes that contain text information Atom-units can be other internal nodes Atom-units can be defined in DTD TF-IDF values are calculated based-on atom-units

16 XIRQL Atom-Units Structured Document Retrieval Principle We should always rank the most specific/probable atom units for answering a query. Example query: xql Document: <chapter> 0.3 XQL <section> 0.5 example </section> <section> 0.8 XQL 0.7 syntax </section> </chapter> Return section, not chapter

17 Structured Document Retrieval Principle Data types: XIRQL suggests vague predicates for different kinds of data types (e.g., person names, locations, dates). It suggests datatype-specific comparison operators (e.g., near, <, >, broader, narrower.) Semantic Roles: search for #persname, XIRQL searches all persons in documents, without specifying their role, regardless of their position in the XML document tree XIRQL Summary Relevance ranking with respect to structure document retrieval principle Recommends datatype-specific operators for different types of data Enable semantic roles

18 Text-Based XML Retrieval Documents are marked up with XML tags journal articles, conference papers, novels, manuals Queries plain text queries, queries with structures (keywords in the title or abstracts) Results System automatically adjust the granularities of the returned results. (e.g., the most specific section about the role of p53 gene for cancer) Considers both coverage and specificity Vector Space Model and XML Vector space model for traditional IR Represent queries and plain documents by vectors in the keyword space. Do not distinguish the keywords in different fields (e.g., title or full text). Calculate similarities between vectors Vector space in XML data Need to capture the structure of an XML document in the vector space.

19 Vector Space Model and XML Flexible queries for XML retrieval Content Only queries (CO) information need of plan text queries, similar to those in traditional information retrieval Content and Structure (CAS) information need of plan text and structure information /book//title Bill Gates or /book//author Bill Gates the structure information can be strict or flexible. (i.e., must from some elements or prefered from some elements) Tree Representation of Queries Book Book Author Bill Gates /book Bill Gates Bill Gates /book//author Bill Gates

20 Vector Space Model and XML Book Book Title Author Title Author Software Bill Gates The plot to get Bill Gates Gary Rivlin Vector Space Model and XML Vector space model for traditional IR System treats the keywords in a document equally; so the two Gates are the same for two documents Vector space in XML data We must distinguish the two occurrences of Gates under different elements Title and Author Index must considers both the contents and the locations of keywords (e.g., different elements)

21 Vector Space Model and XML Vector space in XML data Index must considers both the contents and the locations of keywords (e.g., different elements) To accomplish this, we need to consider the partial trees (structural items) within an XML document. Can we build indexes for the structural items (partials trees)? Vector Space Model and XML Book If we do not allow gap in the tree structures, we can have structural items (partial trees) as Software Bill Gates Title Author Title Author Author Software Bill Gates Software Book Title Bill Book Author Gates Software Bill Gates

22 Vector Space Model and XML Problems of Indexing with Structural items The number of distinct structural items can be very huge. It is not practical to build and store a vector space index with so many dimensions Some possible solutions Build query-time partial vector space Restrict the structural items to a manageable set Vector Space Model and XML Query-time partial vector space Instead of generating all structural items at one time, we can only generate the necessary partial vector space for a specific query (a much smaller set) For a specific query We seek all XML documents with any keyword satisfied the query, build partial vector space from these XML documents The similarity of qualified XML documents and the query can be calculated within the partial vector space

23 Vector Space Model and XML Weights of Structural items (partial trees) Down-weighting for structural items Book Software should have more influence (weight) for book element than Windows, Platform. Title Full Text Software P1 P2 Calculate the weight of a term to an element K levels up by a scaling factor β k, 0<β<1 Windows platform, linux Vector Space Model and XML Weights of Structural items (partial trees) Down-weighting for structural items Book Software should have more influence (weight) for book element than Windows, Platform. Title Full Text Software P1 P2 Calculate the weight of a term to an element K levels up by a scaling factor β k, 0<β<1 Windows platform, linux

24 Vector Space Model and XML Weights of Structural items (partial trees) Down-weighting for structural items Book Weights can also be set for different partial trees. Title Full Text The weights can be predefined Weights can be application oriented Software P1 P2 Weights can be user-specific. Weights can be query-specific. Windows platform, linux Learning issues.. Vector Space Model and XML Other issues of Weights of Structural items (partial trees) Title Software Book Full Text P1 P2 Down-weighting is to use the contents of low-level elements for high-level elements. (e.g., contents of title and full text for book ). Should we also incorporate contents of high-level (or the same level) elements for low-level elemnets? The smoothing strategy Windows platform, linux

25 Vector Space Model and XML Calculating the similarity Vocabulary mismatch of keywords and structures Keyword mismatch has been studied in traditional information retrieval, we can utilize techniques such as query expansion, latent semantic indexing, probabilistic semantic index. Structure mismatch Book Book Book Software Title Full Text Software Software Vector Space Model and XML Calculating the similarity First find all structural items in the query Find all similar match again the vocabulary of structural items It is not a Boolean match, but a similarity match (e.g., 0.9 similarity score with an item) Retrieve all documents/elements with that structural item, compute the cosine similarity etc.

26 Vector Space Model and XML Problems with the vector space model What IDF value? We cannot use a corpus-wide IDF value. The IDF value should be element-specific. But do we need to incorporate the IDF factor of high-level same-level elements? For heterogeneous XML documents We do not exactly know the mapping the schemas. Do we need schema mapping? How can we deal with uncertainty of schema mapping? INEX: Benchmark for text-based XML Retrieval INEX: INitiative for the Evaluation of XML Retrieval The analog of TREC (Text Retrieval Conference) for standard unstructured information retrieval Provide testbed of Set of XML documents, plain queries (content-only queries) and structured queries (with XML structure) A set of retrieval tasks INEX : Mainly organized by people from Europe. It has attracted many participants from universities and big companies from all over the world

27 INEX: Benchmark for text-based XML Retrieval Ad-hoc XML Retrieval Task Each system index a set of XML documents For a set of queries (content-only, content and structure), system convert queries into internal representation In response, each system returns not documents, but most relevant elements within documents Evaluation metrics The retrieved elements are evaluated on two measures: Relevance how relevant is the retrieved element Coverage is the retrieved element too specific, too general or just fine There are scales for the measures, then are turned into precision/recall measures INEX: Benchmark for text-based XML Retrieval Ad-hoc XML Retrieval Task 12,107 articles from IEEE Computer Society publications 494 Megabytes Average article: 1,532 XML nodes/elements Average node/element depth=7

28 INEX: Benchmark for text-based XML Retrieval Relevance: Relevance assessed on a scale from Irrelevant (scoring 0) to Highly Relevant (scoring 3) Coverage No Coverage (N), too general (L), too specific (S), Exact (E) So every element returned by each engine has ratings from {0,1,2,3} {N,S,L,E} INEX: Benchmark for text-based XML Retrieval Define scores: f strict 1 if rel, cov = 3E ( rel, cov) = 0 otherwise f generalized ( rel,cov) = if rel,cov if rel,cov if if rel,cov = 3E rel,cov { 2E,3L,3S} { 1E,2L,2S} { 1S,1 L} if rel,cov = 0N.

29 INEX: Benchmark for text-based XML Retrieval Heterogeneous XML retrieval task: The adhoc track in INEX has dealt with a single DTD of one type of type (computer science journal aritcles) In real-wordl environments, XML retrieval must deal with different DTDs, different genres of data and widely varying topical content Problems: What methods can be used to map structural criteria onto other DTDs? Should mappings focus on element names or also deal with element content or semantic? INEX: Benchmark for text-based XML Retrieval

30 XML Information Retrieval: Outline Basic Concepts of Information Retrieval: Semi-Structure Data XML, Examples, Application XML Search XQuery XIRQL Text-Based XML Retrieval Vector-space model INEX XML Resources - XML resources at W3C Jan-Marco Bremer s publications on xml and ir: Norbert Fuhr and Kai Grossjohann. XIRQL, SIGIR 2001 INEX: Chris Manning: Introduction to Information Retrieval Some contents of the slides are based on above materials