Introduction to Information Retrieval

Transcription

1 Introduction to Information Retrieval WS 2008/ Information Systems Group Mohammed AbuJarour

2 Contents 2 Basics of Information Retrieval (IR) Foundations: extensible Markup Language (XML) Probability & Statistics

3 What is IR? 3 IR by examples: Credit card example. Searching for a book or paper in a library system. Use Google to find a restaurant in Potsdam. Look through a product catalog to find an item. Browse a movie catalog to find an interesting movie.

4 What is IR? 4 IR by definition (scientifically): Representation of information items Storage of information items Organization of information items Access to information items Characterization of the User Information Need is not simple problem: Example: Find all web pages of researchers who studied in Germany and participated at least in 3 EU-funded projects and has been doing research in IR for more than 10 years! This description could not be used directly to get user s information need. Translated into query: typically a set of keywords. IR system (query) relevant information.

5 Data vs. Information Retrieval 5 Data Retrieval Deals with data that has well-defined structure and semantics. Clearly defined conditions, like regular expressions, relational algebra. Information Retrieval Deals with natural language text that is usually not well structured and could be semantically ambiguous. A set of keywords or terms. Each element in the result must satisfy the conditions in the query. An element in the result might be inaccurate with a small number of errors. Example: Name Grade Major Michael B Physics Martin C Mathematics John A Bioinformatik

6 Unstructured (text) vs. Structured (database) Data in 1996 and Unstructured Structured

7 The World Wide Web 7 Huge amount of information. Unusual and diverse documents, e.g., HTML, XHTML, XML, Multimedia... etc. Unusual and diverse users, queries, information needs. (Number of webpages) Size in billion webpages GYWA = Sorted on Google, Yahoo!, Windows Live Search (Msn Search) and Ask YGWA = Sorted on Yahoo!, Google, Windows Live Search (Msn Search) and Ask

8 The Retrieval Process 8 User need User Interface Text Text Text Operations User feedback Query Logical view Query Operations Logical view Inverted file Indexing DB Manager Module Searching Index Retrieved docs Text Database Ranked docs Ranking

9 Basic Concepts in IR 9 Documents: whatever units we have decided to build a retrieval system over, e.g., web page, XML file, pdf file, article, paper, book chapter, product... etc. Collection (Corpus): the group of documents over which we perform retrieval. Information need: the topic about which the user desires to know more. Query: what the user conveys to the computer (system) in an attempt to communicate the information need. Relevance: the degree to which the user perceives a document as containing information of value with respect to his personal information need. DocID: the unique serial number for each document in the collection.

10 Basic Concepts in IR 10 The effectiveness of IR System: the quality of it search results. Precision (Präzision): the fraction of the returned results that are relevant to the information need. Recall (Ausbeute): the fraction of the relevant documents in the collection that were returned by the system.

11 Basic Concepts in IR 10 The effectiveness of IR System: the quality of it search results. Precision (Präzision): the fraction of the returned results that are relevant to the information need. Recall (Ausbeute): the fraction of the relevant documents in the collection that were returned by the system. Collection

12 Basic Concepts in IR 10 The effectiveness of IR System: the quality of it search results. Precision (Präzision): the fraction of the returned results that are relevant to the information need. Recall (Ausbeute): the fraction of the relevant documents in the collection that were returned by the system. Collection Relevant

13 Basic Concepts in IR 10 The effectiveness of IR System: the quality of it search results. Precision (Präzision): the fraction of the returned results that are relevant to the information need. Recall (Ausbeute): the fraction of the relevant documents in the collection that were returned by the system. Collection Relevant Retrieved

14 Basic Concepts in IR 10 The effectiveness of IR System: the quality of it search results. Precision (Präzision): the fraction of the returned results that are relevant to the information need. Recall (Ausbeute): the fraction of the relevant documents in the collection that were returned by the system. Collection Relevant Retrieved Relevant Retrieved

15 Basic Concepts in IR 10 The effectiveness of IR System: the quality of it search results. Precision (Präzision): the fraction of the returned results that are relevant to the information need. Recall (Ausbeute): the fraction of the relevant documents in the collection that were returned by the system. Collection Relevant Retrieved Relevant Retrieved

16 Basic Concepts in IR 10 The effectiveness of IR System: the quality of it search results. Precision (Präzision): the fraction of the returned results that are relevant to the information need. Recall (Ausbeute): the fraction of the relevant documents in the collection that were returned by the system. Collection Relevant Retrieved Relevant Retrieved

17 Contents 11 Basics of Information Retrieval (IR) Foundations: extensible Markup Language (XML) Probability & Statistics

18 extensible Markup Language (XML) 12 Overview of XML: XML was designed to carry/ transfer data, not to display data. XML tags are not predefined. You must define your own tags. XML is designed to be self-descriptive. XML documents may conform to well-defined schemata, e.g., DTD, XSD. XML is a W3C Recommendation. Example:

19 extensible Markup Language (XML) 13 Elements and Relationships

20 extensible Markup Language (XML) 13 Elements and Relationships

21 extensible Markup Language (XML) 13 Elements and Relationships

22 extensible Markup Language (XML) 13 Elements and Relationships

23 extensible Markup Language (XML) 14 Modeling XML Documents Modeled as trees. Tree traversal algorithms: Preorder. Inorder. Postorder. Pre + Post: unique identification of nodes.

24 extensible Markup Language (XML) 14 Modeling XML Documents Modeled as trees. Tree traversal algorithms: Preorder. Inorder. Postorder. Pre + Post: unique identification of nodes

25 extensible Markup Language (XML) 14 Modeling XML Documents Modeled as trees. Tree traversal algorithms: Preorder. Inorder. Postorder. Pre + Post: unique identification of nodes.

26 extensible Markup Language (XML) 14 Modeling XML Documents Modeled as trees. Tree traversal algorithms: Preorder. Inorder. Postorder. Pre + Post: unique identification of nodes

27 extensible Markup Language (XML) 14 Modeling XML Documents Modeled as trees. Tree traversal algorithms: Preorder. Inorder. Postorder. Pre + Post: unique identification of nodes.

28 extensible Markup Language (XML) 14 Modeling XML Documents Modeled as trees. Tree traversal algorithms: Preorder. Inorder. Postorder. Pre + Post: unique identification of nodes

29 extensible Markup Language (XML) 15 XPath is a language for finding information in an XML document. XPath is used to navigate through elements and attributes in an XML document. What is XPath? XPath is a syntax for defining parts of an XML document. XPath uses path expressions to navigate in XML documents. XPath contains a library of standard functions. XPath is a W3C recommendation XPath uses path expressions to select nodes or node-sets in an XML document. The node is selected by following a path or steps.

30 extensible Markup Language (XML) 16 Examples: XPath Syntax XPATH /bookstore Description Selects the root element bookstore bookstore/book Selects all book elements that are children of bookstore //book Selects all book elements no matter where they are in the document bookstore//book Selects all book elements that are descendant of the bookstore element, no matter where they are under the bookstore element Selects all attributes that are named lang /bookstore/ book[price>35.00]/ title Selects all the title elements of the book elements of the bookstore element that have a price element with a value greater than //* Selects all elements in the document

31 extensible Markup Language (XML) 17 XPath expression consists of a sequence of one or more location steps. //descendant::book [position()=1] / child::title [contains ( text(), XML )].

32 extensible Markup Language (XML) 17 XPath expression consists of a sequence of one or more location steps. //descendant::book [position()=1] / child::title [contains ( text(), XML )]. Axis Name

33 extensible Markup Language (XML) 17 XPath expression consists of a sequence of one or more location steps. //descendant::book [position()=1] / child::title [contains ( text(), XML )]. Axis Name Node Test

34 extensible Markup Language (XML) 17 XPath expression consists of a sequence of one or more location steps. //descendant::book [position()=1] / child::title [contains ( text(), XML )]. Axis Name Node Test Predicate Expression

35 extensible Markup Language (XML) 17 XPath expression consists of a sequence of one or more location steps. //descendant::book [position()=1] / child::title [contains ( text(), XML )]. Axis Name Node Test Predicate Expression Location step

36 extensible Markup Language (XML) 17 XPath expression consists of a sequence of one or more location steps. //descendant::book [position()=1] / child::title [contains ( text(), XML )]. Axis Name Node Test Predicate Expression Predicates & Functions Location step

37 extensible Markup Language (XML) 17 XPath expression consists of a sequence of one or more location steps. //descendant::book [position()=1] / child::title [contains ( text(), XML )]. Axis Name Node Test Predicate Expression Predicates & Functions Location step Location step

38 extensible Markup Language (XML) 18 What is XQuery? XQuery is the language for querying XML data. XQuery for XML is like SQL for databases. XQuery is built on XPath expressions. XQuery is a W3C Recommendation Example: XQuery: for $x in doc("books.xml")/bookstore/book where $x/price>30 order by $x/title return $x/title Result: <title lang="eng"> Learning XML</title>

39 Contents 19 Basics of Information Retrieval (IR) Foundations: extensible Markup Language (XML) Probability & Statistics

40 Basics from Probability Theory 20 A probability space is a triple(ω, E, P) with a set Ω of elementary events (sample space), a family E (events) of subsets of Ω with Ω E which is closed under,, and with a countable number of operands, Note: with finite Ω usually E=2 Ω. a probability measure P: E [0,1] with P[Ω]=1 and P[ i Ai] = Σi P[Ai] for countably many, pairwise disjoint Ai. Properties of P: P[A] + P[ A] = 1 P[A B] = P[A] + P[B] P[A B] P[ ] = 0 (null / impossible event) P[Ω] = 1 (true / certain event)

41 Basics from Probability Theory Example 21 Drawing one playing card: Ω of elementary events={s,h,d,c} E (events)= {, S, H, D, C, SH, SD, SC, HD, HC, DC, SHD, SHC, HDC, SDC, SHDC} P[S] + P[ S] = ¼+¾= 1 P[H D] = P[H] + P[D] P[H D] = ¼+¼=½ P[ ] = 0 (null / impossible event) P[SHDC] = ¼+¼+¼+¼=1 Note: SHDC means S H D C.

42 Independence and Conditional Probabilities 22 Two events A, B of a probability space are independent if P[A B] = P[A] P[B]. A finite set of events A={A1,..., An} is independent if for every subset S A the equation holds. The conditional probability P[A B] of A under the condition (hypothesis) B is defined as:

43 Total Probability and Bayes Theorem 23 Total probability theorem: For a partitioning of Ω into events B1,..., Bn: Bayes theorem: P[A B] is called posterior probability. P[A] is called prior probability.

44 Total Probability and Bayes Theorem Example 24 M: a man is chosen. E: the one chosen is employed. P[M] = 500/900= 5/9 P[E] = 600/900 = 2/3 P[M E]= 460/900 = 46/90 Male Female Total Employed Unemployed Total P[M E] = P[M E]/P[E]= 460/600 = 23/30 P[E M] = P[M E]XP[E]/P[M] = (23/30 X 2/3) / (5/9) = 23/25

45 References 25 Baeza-Yates, R. A. and Ribeiro-Neto, B. 1999: Modern Information Retrieval. Addison-Wesley Longman Publishing Co., Inc. Christopher D. Manning, Prabhakar Raghavan, Hinrich Schütze: Introduction to Information Retrieval, Cambridge University Press, David A. Grossman, Ophir Frieder: Information Retrieval: Algorithms and Heuristics, Springer, 2004 Walpole, Myers, Myers, Ye: Probability and Statistics for Engineers and Scientists. Prentice-Hall, Seventh Edition,

46 The End 26 Questions?