Relational Approach. Problem Definition

Size: px

Start display at page:

Download "Relational Approach. Problem Definition"

Horace Ramsey
5 years ago
Views:

1 Relational Approach (COSC 416) Nazli Goharian Slides are mostly based on Information Retrieval Algorithms and Heuristics, Grossman, Frieder Grossman, Frieder 2002, Problem Definition Three conceptual data types: Structured: Objective (absolute) correctness; perfect accuracy; efficiency only issue (e.g.; relational database) Unstructured: Subjective (relative) correctness; accuracy & efficiency trade-off (e.g.; documents, images, video, sound). Semi-structured: Combination of structured and unstructured (e.g.; XML) Problem: Until recently, state of the art individually queried each type specific system and merged answers. Since the early 1990 s our IR Lab and its predecessors work has been to seamlessly integrate the searching of multiple data types. Grossman, Frieder 2002,

2 IR as application of RDBMS Typical IR functionality may be accommodated: Boolean query Relevance Ranking multiple similarity measures Proximity Search Advantages are: Integration Free parallel processing of IR Leverages work done to optimize DBMS Portability Grossman, Frieder 2002, Relational Inverted Index All inverted index entries <term> <list of documents> e.g., vehicle D1, D3, D4 results in: term vehicle vehicle vehicle docid D1 D3 D4 Grossman, Frieder 2002,

3 Text Retrieval Conference (TREC) Sample Document <DOC> <DOCNO> AP </DOCNO> <FILEID>AP-NR EST</FILEID> <FIRST>u i BC-Japan-Stocks </FIRST> <SECOND>BC-Japan-Stocks,0026</SECOND> <HEAD>Stocks Up In Tokyo</HEAD> <DATELINE>TOKYO (AP) </DATELINE> <TEXT> The Nikkei Stock Average closed at 29, points up points on the Tokyo Stock Exchange Wednesday. </TEXT> </DOC> Grossman, Frieder 2002, Relational Document Representation DOCUMENT docid docname headline dateline 28 AP Stocks Up In Tokyo TOKYO (AP) INDEX docid termcnt term 28 1 nikkei 28 2 stock 28 1 average 28 1 closed 28 2 points 28 1 up 28 1 tokyo 28 1 exchange 28 1 wednesday TERM term idf average 1.08 closed 1.08 exchange 1.00 nikkei 2.07 points 1.23 stock 1.00 tokyo 1.58 up 0.30 wednesday 0.60 Grossman, Frieder 2002,

4 Simplistic Models: Keyword and Boolean Searches Grossman, Frieder 2002, Keyword search Relational Approach: Keyword Search select i.docid from INDEX i, QUERY q where i.term = q.term Keyword search with stop word list select i.docid from INDEX i, QUERY q, STOPLIST s where (i.term = q.term) and (i.term <> s.term) Grossman, Frieder 2002,

5 Relational Approach: Boolean Search OR query select docid select docid from INDEX from INDEX where term = term1 where term = term1 OR union term = term2 OR select docid term = term3 OR from INDEX... where term = term2 term = termn union select docid from INDEX where term = term3... union select docid from INDEX where term = termn Grossman, Frieder 2002, Relational Approach: Boolean Search AND query select docid select docid from INDEX from INDEX a, INDEX b, INDEX c,... INDEX N where term = term1 where a.term = term1 AND intersect b.term = term2 AND select docid c.term = term3 AND from INDEX... where term = term2 n.term = termn AND intersect a.docid = b.docid AND select docid b.docid = c.docid AND from INDEX... where term = term3 N-1.docID = N.docID... intersect select docid from INDEX where term = termn Grossman, Frieder 2002,

6 Fixed Join-Count AND Queries Find all documents that contain all of the terms found in the QUERY relation: select i.docid from INDEX i, QUERY q where i.term = q.term group by i.docid having count (distinct (i.term)) = select count(distinct (term)) from QUERY Grossman, Frieder 2002, TAND Queries Find all documents that contain at least X of the terms found in the QUERY relation: select i.docid from INDEX i, QUERY q where i.term = q.term group by i.docid having count (distinct (i.term)) >= X Grossman, Frieder 2002,

7 Relevance Ranking: Vector Space and Probabilistic Models Grossman, Frieder 2002, Relational Document Representation (Single Term Processing) DOCUMENT docid docname headline dateline 28 AP Stocks Up In Tokyo TOKYO (AP) INDEX docid termcnt term 28 1 nikkei 28 2 stock 28 1 average 28 1 closed 28 2 points 28 1 up 28 1 tokyo 28 1 exchange 28 1 wednesday TERM term idf average 1.08 closed 1.08 exchange 1.00 nikkei 2.07 points 1.23 stock 1.00 tokyo 1.58 up 0.30 wednesday 0.60 Grossman, Frieder 2002,

8 Relational Query Representation ORIGINAL QUERY: nikkei stock exchange american stock exchange QUERY Term tf nikkei 1 Stock 2 exchange 2 american 1 Grossman, Frieder 2002, Vector Space Model Term Frequency (tf ik ): number of occurrences of term t k in document i Document Frequency (df j ): number of documents which contain t j Inverse Document Frequency (idf j ): log(d/df j ) where d is the total number of documents Notes: idf is a measure of uniqueness of a term across the collection tf is the frequency of a term in a given document Grossman, Frieder 2002,

9 Similarity Coefficients Several similarity coefficients based on the query vector X and the document vector Y are defined: Inner Prod uct x y Cosine Coefficient t i= 1 i Grossman, Frieder 2002, i t xiyi i= 1 t t 2 xi 2 yi i= 1 i= 1 Relevance Ranking: SQL for VSM dot product List all documents in the order of their similarity coefficient where the coefficient is computed using the dot product. SQL: (Query Weight * Document Weight) SELECT d.docid, d.docname, SUM(q.termcnt * t.idf * i.termcnt * t.idf) FROM QUERY q, INDEX i, TERM t, DOCUMENT d WHERE q.term = i.term AND q.term = t.term AND i.docid = d.docid GROUP BY d.docid, docname ORDER BY 3 DESC Grossman, Frieder 2002,

10 Relevance Ranking: SQL for Probabilistic Retrieval num_ terms log ( numdocs dfi ) tfid ( df ) (.75 doclength avgdoclength) i= 1 i / + tf id qtf SELECT d.docid, d.docname, SUM( LOG(((NumDocs - t.df) + 0.5) / (t.df + 0.5)) * ((2.2*i.tf) / (.3 + ((.75 * d.doclen)/avgdoclen) + i.tf)) * q.termcnt ) FROM INDEX i, TERM t, DOCUMENT d, QUERY q WHERE i.term = t.term AND i.docid = d.docid AND t.term = q.term GROUP BY d.docid, d.docname ORDER BY 3; Grossman, Frieder 2002, Relational Document Representation (Phrase Processing) DOCUMENT docid docname headline dateline 28 AP Stocks Up In Tokyo TOKYO (AP) INDEX docid termcnt phrase 28 1 nikkei stock 28 1 stock average 28 1 average closed 28 1 points up 28 1 tokyo stock 28 1 stock exchange 28 1 exchange wednesday PHRASE phrase idf average closed 2.49 exchange Wednesday 3.33 nikkei stock 2.14 points up 2.61 stock average 2.14 stock exchange 1.34 tokyo stock 2.10 Grossman, Frieder 2002,

11 Simple Phrase Parsing Simple phrase parser with the following rules Phrases do not include stop terms Phrases do not span across symbols Example: The Nikkei Stock Average closed at 29, points up points, on the Tokyo Stock Exchange Wednesday. Phrases: nikkei stock stock average average closed points up tokyo stock stock exchange exchange wednesday Grossman, Frieder 2002, Relational Document Representation (Proximity Search) DOCUMENT docid docname headline dateline 28 AP Stocks Up In Tokyo TOKYO (AP) INDEX docid term offset 28 nikkei stock average closed points up points tokyo stock exchange wednesday 60 Grossman, Frieder 2002,

12 Caveat: Logical Design versus Physical Implementation While the design shown represents the replication of the document identifier, in the physical implementation, clustered tables are actually used. That is, attribute values that are logically repeated many times are physically clustered by the attribute value to eliminate the replication storing only one copy for each unique attribute value. (Note clustered tables in Oracle 8) The I/O to retrieve a posting list is achieved via a grouped block read as opposed to retrieval across distributed storage. Grossman, Frieder 2002, 2010 Clustered Tables: Posting List Processing term docid tf stock { ( 1 1 ), ( 28 2 ), ( ), ( ), ( ), ( ), ( ), ( ), ( ) } Grossman, Frieder 2002,

13 Clustered Tables: Relevance Feedback Processing TRADITIONAL docid termcnt phrase 28 1 nikkei 28 2 stock 28 1 average 28 1 closed 28 2 points 28 1 up 28 1 tokyo 28 1 exchange 28 1 wednesday Grossman, Frieder 2002, 2010 CLUSTERED docid termcnt phrase 28 { ( 1 nikkei ), ( 2 stock ), ( 1 average ), ( 1 closed ), ( 2 points ), ( 1 up ), ( 1 tokyo ), ( 1 exchange ), ( 1 wednesday ) } Relational XML Approach: Architecture XML Collection Semistructured Query SQL Query DB Database Results Grossman, Frieder 2002,

14 XML Search XML provides tags that allow both structured and unstructured data to be represented in the same XML document. Frequently used as a common representation for a variety of document formats. Grossman, Frieder 2002, Introduction XML: Extensible Markup Language Defined by the WWW Consortium (W3C) Derived from SGML (Standard Generalized Markup Language), but simpler to use than SGML Documents have tags giving extra information about sections of the document E.g. <title> XML </title> <slide> Introduction </slide> Extensible, unlike HTML Users can add new tags, and separately specify how the tag should be handled for display A wide variety of tools is available for parsing, browsing and querying XML documents/data Silberschatz, Korth, Sudarshan 14

15 XML Introduction (Cont.) Tags make data (relatively) self-documenting E.g. <university> <department> <dept_name> Comp. Sci. </dept_name> <building> Taylor </building> <budget> </budget> </department> <course> <course_id> CS-101 </course_id> <title> Intro. to Computer Science </title> <dept_name> Comp. Sci </dept_name> <credits> 4 </credits> </course> </university> Silberschatz, Korth, Sudarshan Structure of XML Data Tag: label for a section of data Element: section of data beginning with <tagname> and ending with matching </tagname> Elements must be properly nested Proper nesting <course> <title>. </title> </course> Improper nesting <course> <title>. </course> </title> Formally: every start tag must have a unique matching end tag, that is in the context of the same parent element. Every document must have a single top-level element Silberschatz, Korth, Sudarshan 15

16 Example of Nested Elements <purchase_order> <identifier> P-101 </identifier> <purchaser>. </purchaser> <itemlist> <item> <identifier> RS1 </identifier> <description> Atom powered rocket sled </description> <quantity> 2 </quantity> <price> </price> </item> <item> <identifier> SG2 </identifier> <description> Superb glue </description> <quantity> 1 </quantity> <unit-of-measure> liter </unit-of-measure> <price> </price> </item> </itemlist> </purchase_order> Silberschatz, Korth, Sudarshan Tree Model of XML Data An XML document is modeled as a tree, with nodes corresponding to elements and attributes Element nodes have child nodes, which can be attributes or subelements Text in an element is modeled as a text node child of the element Children of a node are ordered according to their order in the XML document Element and attribute nodes (except for the root node) have a single parent, which is an element node The root node has a single child, which is the root element of the document Silberschatz, Korth, Sudarshan 16

17 Relational XML Approach: Storage <book> <author> John Smith </author> <title> Colonial America </title> <publisher state= NY > NY Publishing </publisher> <book> book author title publisher (E) (E) (E) State= NY (A) Grossman, Frieder 2002, Relational XML Approach: Storage <book> <author> John Smith </author> <title> Colonial America </title> <publisher state= NY > NY Publishing </publisher> <book> pinndx pinndxnum parent tagtype tagpath atrname value 1 0 E 1 1 NULL 2 1 E 2 1 John Smith 3 1 E 3 1 Colonial America 4 1 E 4 1 NY Publishing 5 4 A 4 2 NY tagpath value atrname value 1 [book] 1 NULL 2 [book,author] 2 state 3 [book,title] 4 [book,publisher] atrnametbl tagpathtbl Grossman, Frieder 2002,

18 XQuery XQuery is a general purpose query language for XML data Currently being standardized by the World Wide Web Consortium (W3C) The textbook description is based on a January 2005 draft of the standard. The final version may differ, but major features likely to stay unchanged. XQuery is derived from the Quilt query language, which itself borrows from SQL, XQL and XML-QL XQuery uses a for let where order by result syntax for SQL from where SQL where order by SQL order by result SQL select let allows temporary variables, and has no equivalent in SQL Silberschatz, Korth, Sudarshan Relational XML Approach: Translation XQuery let $item := /order/item where $item[customer_id=3 and credit_card/type= VISA ] order by $item/id Return <item> <id>{$item/id}</id> <price>{$item/price}</price> </item> SQL SELECT DISTINCT q3.value, q3.pinndxnum, q4.value, q4.pinndxnum FROM pinndx q0, pinndx q1, pinndx q2, pinndx q2_1, pinndx q3, pinndx q4 WHERE q0.tagpath=1 AND q1.nvalue=3 AND q1.tagpath=4 AND q1.parent=q0.pinndxnum AND q2.tagpath=5 AND q2.value="visa" AND q2.parent=q2_1.pinndxnum AND q2_1.parent=q0.pinndxnum AND q3.tagpath=7 AND q3.parent=q0.pinndxnum AND q4.tagpath=8 AND q4.parent=q0.pinndxnum ORDER BY q4.nvalue Grossman, Frieder 2002,

Relational Approach. Problem Definition

Relational Approach. Problem Definition Relational Approach (COSC 488) Nazli Goharian nazli@cs.georgetown.edu Slides are mostly based on Information Retrieval Algorithms and Heuristics, Grossman & Frieder 1 Problem Definition Three conceptual