Introduction to Information Retrieval

Size: px
Start display at page:

Download "Introduction to Information Retrieval"

Transcription

1 Introduction Inverted index Processing Boolean queries Course overview Introduction to Information Retrieval IIR 1: Boolean Retrieval Hinrich Schütze Institute for Natural Language Processing, Universität Stuttgart Schütze: Boolean Retrieval 1 / 55

2 Introduction Inverted index Processing Boolean queries Course overview Definition of information retrieval Information retrieval (IR) is finding material (usually documents) of an unstructured nature (usually text) that satisfies an information need from within large collections (usually stored on computers). Schütze: Boolean Retrieval 4 / 55

3 Introduction Inverted index Processing Boolean queries Course overview Boolean retrieval Queries are Boolean expressions, e.g., Caesar and Brutus The seach engine returns all documents that satisfy the Boolean expression. Does Google use the Boolean model? Schütze: Boolean Retrieval 7 / 55

4 Introduction Inverted index Processing Boolean queries Course overview Outline 1 Introduction 2 Inverted index 3 Processing Boolean queries 4 Course overview Schütze: Boolean Retrieval 8 / 55

5 Introduction Inverted index Processing Boolean queries Course overview Unstructured data in 1650 Which plays of Shakespeare contain the words Brutus and Caesar, but not Calpurnia? One could grep all of Shakespeare s plays for Brutus and Caesar, then strip out lines containing Calpurnia. Why is grep not the solution? Schütze: Boolean Retrieval 10 / 55

6 Introduction Inverted index Processing Boolean queries Course overview Unstructured data in 1650 Which plays of Shakespeare contain the words Brutus and Caesar, but not Calpurnia? One could grep all of Shakespeare s plays for Brutus and Caesar, then strip out lines containing Calpurnia. Why is grep not the solution? Slow (for large collections) not Calpurnia is non-trivial Other operations (e.g., find the word Romans near countryman) not feasible Ranked retrieval (best documents to return) focus of later lectures, but not this one Schütze: Boolean Retrieval 10 / 55

7 Introduction Inverted index Processing Boolean queries Course overview Term-document incidence matrix Anthony Julius The Hamlet Othello Macbeth... and Caesar Tempest Cleopatra Anthony Brutus Caesar Calpurnia Cleopatra mercy worser Entry is 1 if term occurs. Example: Calpurnia occurs in Julius Caesar. Entry is 0 if term doesn t occur. Example: Calpurnia doesn t occur in The tempest. Schütze: Boolean Retrieval 11 / 55

8 Introduction Inverted index Processing Boolean queries Course overview Incidence vectors So we have a 0/1 vector for each term. To answer the query Brutus and Caesar and not Calpurnia: Schütze: Boolean Retrieval 12 / 55

9 Introduction Inverted index Processing Boolean queries Course overview Incidence vectors So we have a 0/1 vector for each term. To answer the query Brutus and Caesar and not Calpurnia: Take the vectors for Brutus, Caesar, and Calpurnia Complement the vector of Calpurnia Do a (bitwise) and on the three vectors and and = Schütze: Boolean Retrieval 12 / 55

10 Introduction Inverted index Processing Boolean queries Course overview 0/1 vector for Brutus Anthony Julius The Hamlet Othello Macbeth... and Caesar Tempest Cleopatra Anthony Brutus Caesar Calpurnia Cleopatra mercy worser Schütze: Boolean Retrieval 13 / 55

11 Introduction Inverted index Processing Boolean queries Course overview Bigger collections Consider N = 10 6 documents, each with about 1000 tokens On average 6 bytes per token, including spaces and punctuation size of document collection is about 6 GB Assume there are M = 500,000 distinct terms in the collection (Notice that we are making a term/token distinction.) Schütze: Boolean Retrieval 15 / 55

12 Introduction Inverted index Processing Boolean queries Course overview Can t build the incidence matrix M = 500, = half a trillion 0s and 1s. But the matrix has no more than one billion 1s. Matrix is extremely sparse. What is a better representations? We only record the 1s. Schütze: Boolean Retrieval 16 / 55

13 Introduction Inverted index Processing Boolean queries Course overview Inverted Index For each term t, we store a list of all documents that contain t. Brutus Caesar Calpurnia }{{}}{{} dictionary postings Schütze: Boolean Retrieval 17 / 55

14 Introduction Inverted index Processing Boolean queries Course overview Inverted index construction 1 Collect the documents to be indexed: Friends, Romans, countrymen. So let it be with Caesar... 2 Tokenize the text, turning each document into a list of tokens: Friends Romans countrymen So... 3 Do linguistic preprocessing, producing a list of normalized tokens, which are the indexing terms: friend roman countryman so... 4 Index the documents that each term occurs in by creating an inverted index, consisting of a dictionary and postings. Schütze: Boolean Retrieval 18 / 55

15 Introduction Inverted index Processing Boolean queries Course overview Outline 1 Introduction 2 Inverted index 3 Processing Boolean queries 4 Course overview Schütze: Boolean Retrieval 25 / 55

16 Introduction Inverted index Processing Boolean queries Course overview Simple conjunctive query (two terms) Consider the query: Brutus AND Calpurnia To find all matching documents using inverted index: 1 Locate Brutus in the dictionary 2 Retrieve its postings list from the postings file 3 Locate Calpurnia in the dictionary 4 Retrieve its postings list from the postings file 5 Intersect the two postings lists 6 Return intersection to user Schütze: Boolean Retrieval 26 / 55

17 Introduction Inverted index Processing Boolean queries Course overview Intersecting two postings lists Brutus Calpurnia Intersection = Schütze: Boolean Retrieval 27 / 55

18 Introduction Inverted index Processing Boolean queries Course overview Intersecting two postings lists Brutus Calpurnia Intersection = Schütze: Boolean Retrieval 27 / 55

19 Introduction Inverted index Processing Boolean queries Course overview Intersecting two postings lists Brutus Calpurnia Intersection = Schütze: Boolean Retrieval 27 / 55

20 Introduction Inverted index Processing Boolean queries Course overview Intersecting two postings lists Brutus Calpurnia Intersection = 2 Schütze: Boolean Retrieval 27 / 55

21 Introduction Inverted index Processing Boolean queries Course overview Intersecting two postings lists Brutus Calpurnia Intersection = 2 Schütze: Boolean Retrieval 27 / 55

22 Introduction Inverted index Processing Boolean queries Course overview Intersecting two postings lists Brutus Calpurnia Intersection = 2 Schütze: Boolean Retrieval 27 / 55

23 Introduction Inverted index Processing Boolean queries Course overview Intersecting two postings lists Brutus Calpurnia Intersection = 2 Schütze: Boolean Retrieval 27 / 55

24 Introduction Inverted index Processing Boolean queries Course overview Intersecting two postings lists Brutus Calpurnia Intersection = 2 31 Schütze: Boolean Retrieval 27 / 55

25 Introduction Inverted index Processing Boolean queries Course overview Intersecting two postings lists Brutus Calpurnia Intersection = 2 31 Schütze: Boolean Retrieval 27 / 55

26 Introduction Inverted index Processing Boolean queries Course overview Intersecting two postings lists Brutus Calpurnia Intersection = 2 31 Schütze: Boolean Retrieval 27 / 55

27 Introduction Inverted index Processing Boolean queries Course overview Intersecting two postings lists Brutus Calpurnia Intersection = 2 31 Schütze: Boolean Retrieval 27 / 55

28 Introduction Inverted index Processing Boolean queries Course overview Intersecting two postings lists Brutus Calpurnia Intersection = 2 31 Schütze: Boolean Retrieval 27 / 55

29 Recap The term vocabulary Skip pointers Phrase queries Recall basic intersection algorithm Brutus Calpurnia Intersection = 2 31 Can we do better? Schütze: The term vocabulary and postings lists 40 / 60

30 Recap The term vocabulary Skip pointers Phrase queries Skip lists Schütze: The term vocabulary and postings lists 42 / 60

31 Recap The term vocabulary Skip pointers Phrase queries Outline 1 Recap 2 The term vocabulary 3 Skip pointers 4 Phrase queries Schütze: The term vocabulary and postings lists 8 / 60

32 Recap The term vocabulary Skip pointers Phrase queries Definitions Word A delimited string of characters as it appears in the text. Term A normalized word (case, morphology, spelling etc); an equivalence class of words. Token An instance of a word or term occurring in a document. Type The same as a term in most cases: an equivalence class of tokens. Schütze: The term vocabulary and postings lists 13 / 60

33 Recap The term vocabulary Skip pointers Phrase queries Recall: Inverted index construction Input: Friends, Romans, countrymen. So let it be with Caesar... Output: friend roman countryman so... Each token is a candidate for a postings entry. What are valid tokens to emit? Schütze: The term vocabulary and postings lists 15 / 60

34 Recap The term vocabulary Skip pointers Phrase queries Stop words stop words = extremely common words which would appear to be of little value in helping select documents matching a user need Examples: a, an, and, are, as, at, be, by, for, from, has, he, in, is, it, its, of, on, that, the, to, was, were, will, with Stop word elimination used to be standard in older IR systems. But you need stop words for phrase queries, e.g. King of Denmark Most web search engines index stop words. Schütze: The term vocabulary and postings lists 29 / 60

35 Recap The term vocabulary Skip pointers Phrase queries Lemmatization Reduce inflectional/variant forms to base form Example: am, are, is be Example: car, cars, car s, cars car Example: the boy s cars are different colors the boy car be different color Lemmatization implies doing proper reduction to dictionary headword form (the lemma). Inflectional morphology (cutting cut) vs. derivational morphology (destruction destroy) Schütze: The term vocabulary and postings lists 32 / 60

36 Recap The term vocabulary Skip pointers Phrase queries Stemming Definition of stemming: Crude heuristic process that chops off the ends of words in the hope of achieving what principled lemmatization attempts to do with a lot of linguistic knowledge. Language dependent Often inflectional and derivational Example for derivational: automate, automatic, automation all reduce to automat Schütze: The term vocabulary and postings lists 33 / 60

37 Recap The term vocabulary Skip pointers Phrase queries Porter stemmer: A few rules Rule Example SSES SS caresses caress IES I ponies poni SS SS caress caress S cats cat Schütze: The term vocabulary and postings lists 35 / 60

38 Introduction to Information Retrieval IIR 6: Scoring, Term Weighting, The Vector Space Model Hinrich Schütze Institute for Natural Language Processing, Universität Stuttgart Schütze: Scoring, term weighting, the vector space model 1 / 53

39 Outline 1 Recap 2 Term frequency 3 tf-idf weighting 4 The vector space Schütze: Scoring, term weighting, the vector space model 9 / 53

40 Ranked retrieval Thus far, our queries have all been Boolean. Documents either match or don t. Good for expert users with precise understanding of their needs and the collection. Also good for applications: Applications can easily consume 1000s of results. Not good for the majority of users. Most users are not capable of writing Boolean queries (or they are, but they think it s too much work). Most users don t want to wade through 1000s of results. This is particularly true of web search. Schütze: Scoring, term weighting, the vector space model 10 / 53

41 Problem with Boolean search: Feast or famine Boolean queries often result in either too few (=0) or too many (1000s) results. Query 1: standard user dlink ,000 hits Query 2: standard user dlink 650 no card found : 0 hits It takes a lot of skill to come up with a query that produces a manageable number of hits. With a ranked list of documents it does not matter how large the retrieved set is. Schütze: Scoring, term weighting, the vector space model 11 / 53

42 Scoring as the basis of ranked retrieval We wish to return in order the documents most likely to be useful to the searcher. How can we rank-order the documents in the collection with respect to a query? Assign a score say in [0, 1] to each document This score measures how well document and query match. Schütze: Scoring, term weighting, the vector space model 12 / 53

43 Query-document matching scores We need a way of assigning a score to a query/document pair. Let s start with a one-term query. If the query term does not occur in the document: score should be 0. The more frequent the query term in the document, the higher the score Schütze: Scoring, term weighting, the vector space model 13 / 53

44 Recall: Binary incidence matrix Anthony Julius The Hamlet Othello Macbeth... and Caesar Tempest Cleopatra Anthony Brutus Caesar Calpurnia Cleopatra mercy worser Each document is represented by a binary vector {0, 1} V. Schütze: Scoring, term weighting, the vector space model 17 / 53

45 From now on, we will use the frequencies of terms Anthony Julius The Hamlet Othello Macbeth... and Caesar Tempest Cleopatra Anthony Brutus Caesar Calpurnia Cleopatra mercy worser Each document is represented by a count vector N V. Schütze: Scoring, term weighting, the vector space model 18 / 53

46 Bag of words model We do not consider the order of words in a document. John is quicker than Mary and Mary is quicker than John are represented the same way. This is called a bag of words model. Schütze: Scoring, term weighting, the vector space model 19 / 53

47 Term frequency tf The term frequency tf t,d of term t in document d is defined as the number of times that t occurs in d. We want to use tf when computing query-document match scores. But how? Schütze: Scoring, term weighting, the vector space model 20 / 53

48 Term frequency tf The term frequency tf t,d of term t in document d is defined as the number of times that t occurs in d. We want to use tf when computing query-document match scores. But how? Raw term frequency is not what we want. A document with 10 occurrences of the term is more relevant than a document with one occurrence of the term. But not 10 times more relevant. Relevance does not increase proportionally with term frequency. Schütze: Scoring, term weighting, the vector space model 20 / 53

49 Log frequency weighting The log frequency weight of term t in d is defined as follows { 1 + log10 tf w t,d = t,d if tf t,d > 0 0 otherwise 0 0, 1 1, 2 1.3, 10 2, , etc. Schütze: Scoring, term weighting, the vector space model 21 / 53

50 Log frequency weighting The log frequency weight of term t in d is defined as follows { 1 + log10 tf w t,d = t,d if tf t,d > 0 0 otherwise 0 0, 1 1, 2 1.3, 10 2, , etc. Score for a document-query pair: sum over terms t in both q and d: matching-score = t q d (1 + log tf t,d) Schütze: Scoring, term weighting, the vector space model 21 / 53

51 Log frequency weighting The log frequency weight of term t in d is defined as follows { 1 + log10 tf w t,d = t,d if tf t,d > 0 0 otherwise 0 0, 1 1, 2 1.3, 10 2, , etc. Score for a document-query pair: sum over terms t in both q and d: matching-score = t q d (1 + log tf t,d) The score is 0 if none of the query terms is present in the document. Schütze: Scoring, term weighting, the vector space model 21 / 53

52 Outline 1 Recap 2 Term frequency 3 tf-idf weighting 4 The vector space Schütze: Scoring, term weighting, the vector space model 22 / 53

53 Document frequency Rare terms are more informative than frequent terms. Consider a term in the query that is rare in the collection (e.g., arachnocentric) A document containing this term is very likely to be relevant. We want a high weight for rare terms like arachnocentric. Schütze: Scoring, term weighting, the vector space model 23 / 53

54 Document frequency Rare terms are more informative than frequent terms. Consider a term in the query that is rare in the collection (e.g., arachnocentric) A document containing this term is very likely to be relevant. We want a high weight for rare terms like arachnocentric. Consider a term in the query that is frequent in the collection (e.g., high, increase, line) A document containing this term is more likely to be relevant than a document that doesn t, but it s not a sure indicator of relevance. For frequent terms, we want positive weights for words like high, increase, and line, but lower weights than for rare terms. Schütze: Scoring, term weighting, the vector space model 23 / 53

55 Document frequency Rare terms are more informative than frequent terms. Consider a term in the query that is rare in the collection (e.g., arachnocentric) A document containing this term is very likely to be relevant. We want a high weight for rare terms like arachnocentric. Consider a term in the query that is frequent in the collection (e.g., high, increase, line) A document containing this term is more likely to be relevant than a document that doesn t, but it s not a sure indicator of relevance. For frequent terms, we want positive weights for words like high, increase, and line, but lower weights than for rare terms. We will use document frequency to factor this into computing the matching score. The document frequency is the number of documents in the collection that the term occurs in. Schütze: Scoring, term weighting, the vector space model 23 / 53

56 idf weight df t is the document frequency, the number of documents that t occurs in. df is an inverse measure of the informativeness of the term. We define the idf weight of term t as follows: idf t = log 10 N df t idf is a measure of the informativeness of the term. Schütze: Scoring, term weighting, the vector space model 24 / 53

57 Examples for idf Compute idf t using the formula: idf t = log 10 1,000,000 df t term df t idf t calpurnia 1 6 animal sunday fly 10,000 2 under 100,000 1 the 1,000,000 0 Schütze: Scoring, term weighting, the vector space model 25 / 53

58 tf-idf weighting The tf-idf weight of a term is the product of its tf weight and its idf weight. w t,d = (1 + log tf t,d ) log N df t Best known weighting scheme in information retrieval Note: the - in tf-idf is a hyphen, not a minus sign! Schütze: Scoring, term weighting, the vector space model 28 / 53

59 Summary: tf-idf Assign a tf-idf weight for each term t in each document d: w t,d = (1 + log tf t,d ) log N df t N: total number of documents Increases with the number of occurrences within a document Increases with the rarity of the term in the collection Schütze: Scoring, term weighting, the vector space model 29 / 53

60 Outline 1 Recap 2 Term frequency 3 tf-idf weighting 4 The vector space Schütze: Scoring, term weighting, the vector space model 31 / 53

61 Binary count weight matrix Anthony Julius The Hamlet Othello Macbeth... and Caesar Tempest Cleopatra Anthony Brutus Caesar Calpurnia Cleopatra mercy worser Each document is now represented by a real-valued vector of tf-idf weights R V. Schütze: Scoring, term weighting, the vector space model 32 / 53

62 Documents as vectors Each document is now represented by a real-valued vector of tf-idf weights R V. So we have a V -dimensional real-valued vector space. Terms are axes of the space. Documents are points or vectors in this space. Very high-dimensional: tens of millions of dimensions when you apply this to a web search engine This is a very sparse vector - most entries are zero. Schütze: Scoring, term weighting, the vector space model 33 / 53

63 Queries as vectors Key idea 1: do the same for queries: represent them as vectors in the space Key idea 2: Rank documents according to their proximity to the query Schütze: Scoring, term weighting, the vector space model 34 / 53

64 How do we formalize vector space similarity? First cut: distance between two points ( = distance between the end points of the two vectors) Euclidean distance? Schütze: Scoring, term weighting, the vector space model 35 / 53

65 How do we formalize vector space similarity? First cut: distance between two points ( = distance between the end points of the two vectors) Euclidean distance? Euclidean distance is a bad idea......because Euclidean distance is large for vectors of different lengths. Schütze: Scoring, term weighting, the vector space model 35 / 53

66 Why distance is a bad idea gossip d 2 1 d 1 q The Euclidean distance of q and d 2 is large although the distribution of terms in the query q and the distribution of terms in the document d 2 are very similar d 3 jealous Schütze: Scoring, term weighting, the vector space model 36 / 53

67 Use angle instead of distance Rank documents according to angle with query Thought experiment: take a document d and append it to itself. Call this document d. Semantically d and d have the same content. The angle between the two documents is 0, corresponding to maximal similarity. The Euclidean distance between the two documents can be quite large. Schütze: Scoring, term weighting, the vector space model 37 / 53

68 From angles to cosines The following two notions are equivalent. Rank documents according to the angle between query and document in decreasing order Rank documents according to cosine(query,document) in increasing order Cosine is a monotonically decreasing function of the angle for the interval [0, 180 ] Schütze: Scoring, term weighting, the vector space model 38 / 53

69 Length normalization How do we compute the cosine? A vector can be (length-) normalized by dividing each of its components by its length here we use the L 2 norm: x 2 = i x2 i This maps vectors onto the unit sphere......since after normalization: x 2 = i x2 i = 1.0 As a result, longer documents and shorter documents have weights of the same order of magnitude. Effect on the two documents d and d (d appended to itself) from earlier slide: they have identical vectors after length-normalization. Schütze: Scoring, term weighting, the vector space model 41 / 53

70 Cosine similarity between query and document cos( q, d) = sim( q, d) = q d q d = V V i=1 q2 i i=1 q id i V i=1 d2 i q i is the tf-idf weight of term i in the query. d i is the tf-idf weight of term i in the document. q and d are the lengths of q and d. This is the cosine similarity of q and d...or, equivalently, the cosine of the angle between q and d. Schütze: Scoring, term weighting, the vector space model 42 / 53

71 Cosine similarity illustrated gossip 1 v(d 1 ) v(q) v(d 2 ) θ v(d 3 ) jealous Schütze: Scoring, term weighting, the vector space model 44 / 53

72 Cosine: Example How similar are the novels? SaS: Sense and Sensibility, PaP: Pride and Prejudice, and WH: Wuthering Heights? term frequencies (counts) term SaS PaP WH affection jealous gossip wuthering Schütze: Scoring, term weighting, the vector space model 45 / 53

73 Cosine: Example term frequencies (counts) term SaS PaP WH affection jealous gossip wuthering log frequency weighting term SaS PaP WH affection jealous gossip wuthering (To simplify this example, we don t do idf weighting.) Schütze: Scoring, term weighting, the vector space model 46 / 53

74 Cosine: Example log frequency weighting term SaS PaP WH affection jealous gossip wuthering log frequency weighting & cosine normalization term SaS PaP WH affection jealous gossip wuthering Schütze: Scoring, term weighting, the vector space model 47 / 53

75 Cosine: Example log frequency weighting term SaS PaP WH affection jealous gossip wuthering log frequency weighting & cosine normalization term SaS PaP WH affection jealous gossip wuthering cos(sas,pap) cos(sas,wh) 0.79 cos(pap,wh) 0.69 Why do we have cos(sas,pap) > cos(sas,wh)? Schütze: Scoring, term weighting, the vector space model 47 / 53

76 Summary: Ranked retrieval in the vector space model Represent the query as a weighted tf-idf vector Represent each document as a weighted tf-idf vector Compute the cosine similarity between the query vector and each document vector Rank documents with respect to the query Return the top K (e.g., K = 10) to the user Schütze: Scoring, term weighting, the vector space model 52 / 53

Information Retrieval

Information Retrieval Information Retrieval Suan Lee - Information Retrieval - 06 Scoring, Term Weighting and the Vector Space Model 1 Recap of lecture 5 Collection and vocabulary statistics: Heaps and Zipf s laws Dictionary

More information

This lecture: IIR Sections Ranked retrieval Scoring documents Term frequency Collection statistics Weighting schemes Vector space scoring

This lecture: IIR Sections Ranked retrieval Scoring documents Term frequency Collection statistics Weighting schemes Vector space scoring This lecture: IIR Sections 6.2 6.4.3 Ranked retrieval Scoring documents Term frequency Collection statistics Weighting schemes Vector space scoring 1 Ch. 6 Ranked retrieval Thus far, our queries have all

More information

Information Retrieval

Information Retrieval Introduction to Information Retrieval Lecture 6-: Scoring, Term Weighting Outline Why ranked retrieval? Term frequency tf-idf weighting 2 Ranked retrieval Thus far, our queries have all been Boolean. Documents

More information

Informa(on Retrieval

Informa(on Retrieval Introduc)on to Informa)on Retrieval CS3245 Informa(on Retrieval Lecture 7: Scoring, Term Weigh9ng and the Vector Space Model 7 Last Time: Index Construc9on Sort- based indexing Blocked Sort- Based Indexing

More information

Information Retrieval

Information Retrieval Information Retrieval Natural Language Processing: Lecture 12 30.11.2017 Kairit Sirts Homework 4 things that seemed to work Bidirectional LSTM instead of unidirectional Change LSTM activation to sigmoid

More information

Informa(on Retrieval

Informa(on Retrieval Introduc)on to Informa)on Retrieval CS3245 Informa(on Retrieval Lecture 7: Scoring, Term Weigh9ng and the Vector Space Model 7 Last Time: Index Compression Collec9on and vocabulary sta9s9cs: Heaps and

More information

FRONT CODING. Front-coding: 8automat*a1 e2 ic3 ion. Extra length beyond automat. Encodes automat. Begins to resemble general string compression.

FRONT CODING. Front-coding: 8automat*a1 e2 ic3 ion. Extra length beyond automat. Encodes automat. Begins to resemble general string compression. Sec. 5.2 FRONT CODING Front-coding: Sorted words commonly have long common prefix store differences only (for last k-1 in a block of k) 8automata8automate9automatic10automation 8automat*a1 e2 ic3 ion Encodes

More information

CS105 Introduction to Information Retrieval

CS105 Introduction to Information Retrieval CS105 Introduction to Information Retrieval Lecture: Yang Mu UMass Boston Slides are modified from: http://www.stanford.edu/class/cs276/ Information Retrieval Information Retrieval (IR) is finding material

More information

Introduction to Information Retrieval

Introduction to Information Retrieval Introduction to Information Retrieval http://informationretrieval.org IIR 1: Boolean Retrieval Hinrich Schütze Institute for Natural Language Processing, University of Stuttgart 2011-05-03 1/ 36 Take-away

More information

Introduction to Information Retrieval

Introduction to Information Retrieval Introduction to Information Retrieval http://informationretrieval.org IIR 1: Boolean Retrieval Hinrich Schütze Center for Information and Language Processing, University of Munich 2014-04-09 Schütze: Boolean

More information

CSE 7/5337: Information Retrieval and Web Search Introduction and Boolean Retrieval (IIR 1)

CSE 7/5337: Information Retrieval and Web Search Introduction and Boolean Retrieval (IIR 1) CSE 7/5337: Information Retrieval and Web Search Introduction and Boolean Retrieval (IIR 1) Michael Hahsler Southern Methodist University These slides are largely based on the slides by Hinrich Schütze

More information

Digital Libraries: Language Technologies

Digital Libraries: Language Technologies Digital Libraries: Language Technologies RAFFAELLA BERNARDI UNIVERSITÀ DEGLI STUDI DI TRENTO P.ZZA VENEZIA, ROOM: 2.05, E-MAIL: BERNARDI@DISI.UNITN.IT Contents 1 Recall: Inverted Index..........................................

More information

Introduction to Information Retrieval

Introduction to Information Retrieval Boolean model and Inverted index Processing Boolean queries Why ranked retrieval? Introduction to Information Retrieval http://informationretrieval.org IIR 1: Boolean Retrieval Hinrich Schütze Institute

More information

Information Retrieval and Organisation

Information Retrieval and Organisation Information Retrieval and Organisation Dell Zhang Birkbeck, University of London 2016/17 IR Chapter 01 Boolean Retrieval Example IR Problem Let s look at a simple IR problem Suppose you own a copy of Shakespeare

More information

The Web document collection

The Web document collection Web Data Management Part 1 Advanced Topics in Database Management (INFSCI 2711) Textbooks: Database System Concepts - 2010 Introduction to Information Retrieval - 2008 Vladimir Zadorozhny, DINS, SCI, University

More information

boolean queries Inverted index query processing Query optimization boolean model September 9, / 39

boolean queries Inverted index query processing Query optimization boolean model September 9, / 39 boolean model September 9, 2014 1 / 39 Outline 1 boolean queries 2 3 4 2 / 39 taxonomy of IR models Set theoretic fuzzy extended boolean set-based IR models Boolean vector probalistic algebraic generalized

More information

Information Retrieval

Information Retrieval Introduction to Information Retrieval CS276 Information Retrieval and Web Search Christopher Manning and Prabhakar Raghavan Lecture 1: Boolean retrieval Information Retrieval Information Retrieval (IR)

More information

Introduction to Information Retrieval

Introduction to Information Retrieval Introduction to Information Retrieval http://informationretrieval.org IIR 1: Boolean Retrieval Hinrich Schütze Institute for Natural Language Processing, Universität Stuttgart 2008.04.22 Schütze: Boolean

More information

INFO 4300 / CS4300 Information Retrieval. slides adapted from Hinrich Schütze s, linked from

INFO 4300 / CS4300 Information Retrieval. slides adapted from Hinrich Schütze s, linked from INFO 4300 / CS4300 Information Retrieval slides adapted from Hinrich Schütze s, linked from http://informationretrieval.org/ IR 1: Boolean Retrieval Paul Ginsparg Cornell University, Ithaca, NY 27 Aug

More information

Information Retrieval CS Lecture 06. Razvan C. Bunescu School of Electrical Engineering and Computer Science

Information Retrieval CS Lecture 06. Razvan C. Bunescu School of Electrical Engineering and Computer Science Information Retrieval CS 6900 Lecture 06 Razvan C. Bunescu School of Electrical Engineering and Computer Science bunescu@ohio.edu Boolean Retrieval vs. Ranked Retrieval Many users (professionals) prefer

More information

Information Retrieval and Text Mining

Information Retrieval and Text Mining Information Retrieval and Text Mining http://informationretrieval.org IIR 1: Boolean Retrieval Hinrich Schütze & Wiltrud Kessler Institute for Natural Language Processing, University of Stuttgart 2012-10-16

More information

Information Retrieval

Information Retrieval Introduction to Information Retrieval Information Retrieval and Web Search Lecture 1: Introduction and Boolean retrieval Outline ❶ Course details ❷ Information retrieval ❸ Boolean retrieval 2 Course details

More information

Information Retrieval

Information Retrieval Introduction to Information Retrieval CS276 Information Retrieval and Web Search Pandu Nayak and Prabhakar Raghavan Lecture 1: Boolean retrieval Information Retrieval Information Retrieval (IR) is finding

More information

Boolean retrieval & basics of indexing CE-324: Modern Information Retrieval Sharif University of Technology

Boolean retrieval & basics of indexing CE-324: Modern Information Retrieval Sharif University of Technology Boolean retrieval & basics of indexing CE-324: Modern Information Retrieval Sharif University of Technology M. Soleymani Fall 2016 Most slides have been adapted from: Profs. Manning, Nayak & Raghavan lectures

More information

Boolean Retrieval. Manning, Raghavan and Schütze, Chapter 1. Daniël de Kok

Boolean Retrieval. Manning, Raghavan and Schütze, Chapter 1. Daniël de Kok Boolean Retrieval Manning, Raghavan and Schütze, Chapter 1 Daniël de Kok Boolean query model Pose a query as a boolean query: Terms Operations: AND, OR, NOT Example: Brutus AND Caesar AND NOT Calpuria

More information

Introduction to Information Retrieval and Boolean model. Reference: Introduction to Information Retrieval by C. Manning, P. Raghavan, H.

Introduction to Information Retrieval and Boolean model. Reference: Introduction to Information Retrieval by C. Manning, P. Raghavan, H. Introduction to Information Retrieval and Boolean model Reference: Introduction to Information Retrieval by C. Manning, P. Raghavan, H. Schutze 1 Unstructured (text) vs. structured (database) data in late

More information

Advanced Retrieval Information Analysis Boolean Retrieval

Advanced Retrieval Information Analysis Boolean Retrieval Advanced Retrieval Information Analysis Boolean Retrieval Irwan Ary Dharmawan 1,2,3 iad@unpad.ac.id Hana Rizmadewi Agustina 2,4 hagustina@unpad.ac.id 1) Development Center of Information System and Technology

More information

INFO 4300 / CS4300 Information Retrieval. slides adapted from Hinrich Schütze s, linked from

INFO 4300 / CS4300 Information Retrieval. slides adapted from Hinrich Schütze s, linked from INFO 4300 / CS4300 Information Retrieval slides adapted from Hinrich Schütze s, linked from http://informationretrieval.org/ IR 6: Index Compression Paul Ginsparg Cornell University, Ithaca, NY 15 Sep

More information

Introducing Information Retrieval and Web Search. borrowing from: Pandu Nayak

Introducing Information Retrieval and Web Search. borrowing from: Pandu Nayak Introducing Information Retrieval and Web Search borrowing from: Pandu Nayak Information Retrieval Information Retrieval (IR) is finding material (usually documents) of an unstructured nature (usually

More information

Introduction to Information Retrieval

Introduction to Information Retrieval Mustafa Jarrar: Lecture Notes on Information Retrieval University of Birzeit, Palestine 2014 Introduction to Information Retrieval Dr. Mustafa Jarrar Sina Institute, University of Birzeit mjarrar@birzeit.edu

More information

Information Retrieval

Information Retrieval Introduction to Information Retrieval CS3245 Information Retrieval Lecture 2: Boolean retrieval 2 Blanks on slides, you may want to fill in Last Time: Ngram Language Models Unigram LM: Bag of words Ngram

More information

Information Retrieval

Information Retrieval Information Retrieval Suan Lee - Information Retrieval - 01 Boolean Retrieval 1 01 Boolean Retrieval - Information Retrieval - 01 Boolean Retrieval 2 Introducing Information Retrieval and Web Search -

More information

Unstructured Data Management. Advanced Topics in Database Management (INFSCI 2711)

Unstructured Data Management. Advanced Topics in Database Management (INFSCI 2711) Unstructured Data Management Advanced Topics in Database Management (INFSCI 2711) Textbooks: Database System Concepts - 2010 Introduction to Information Retrieval - 2008 Vladimir Zadorozhny, DINS, SCI,

More information

Models for Document & Query Representation. Ziawasch Abedjan

Models for Document & Query Representation. Ziawasch Abedjan Models for Document & Query Representation Ziawasch Abedjan Overview Introduction & Definition Boolean retrieval Vector Space Model Probabilistic Information Retrieval Language Model Approach Summary Overview

More information

Boolean retrieval & basics of indexing CE-324: Modern Information Retrieval Sharif University of Technology

Boolean retrieval & basics of indexing CE-324: Modern Information Retrieval Sharif University of Technology Boolean retrieval & basics of indexing CE-324: Modern Information Retrieval Sharif University of Technology M. Soleymani Fall 2013 Most slides have been adapted from: Profs. Manning, Nayak & Raghavan (CS-276,

More information

Part 2: Boolean Retrieval Francesco Ricci

Part 2: Boolean Retrieval Francesco Ricci Part 2: Boolean Retrieval Francesco Ricci Most of these slides comes from the course: Information Retrieval and Web Search, Christopher Manning and Prabhakar Raghavan Content p Term document matrix p Information

More information

Indexing. Lecture Objectives. Text Technologies for Data Science INFR Learn about and implement Boolean search Inverted index Positional index

Indexing. Lecture Objectives. Text Technologies for Data Science INFR Learn about and implement Boolean search Inverted index Positional index Text Technologies for Data Science INFR11145 Indexing Instructor: Walid Magdy 03-Oct-2017 Lecture Objectives Learn about and implement Boolean search Inverted index Positional index 2 1 Indexing Process

More information

Search: the beginning. Nisheeth

Search: the beginning. Nisheeth Search: the beginning Nisheeth Interdisciplinary area Information retrieval NLP Search Machine learning Human factors Outline Components Crawling Processing Indexing Retrieval Evaluation Research areas

More information

Classic IR Models 5/6/2012 1

Classic IR Models 5/6/2012 1 Classic IR Models 5/6/2012 1 Classic IR Models Idea Each document is represented by index terms. An index term is basically a (word) whose semantics give meaning to the document. Not all index terms are

More information

Boolean retrieval & basics of indexing CE-324: Modern Information Retrieval Sharif University of Technology

Boolean retrieval & basics of indexing CE-324: Modern Information Retrieval Sharif University of Technology Boolean retrieval & basics of indexing CE-324: Modern Information Retrieval Sharif University of Technology M. Soleymani Fall 2015 Most slides have been adapted from: Profs. Manning, Nayak & Raghavan lectures

More information

Introduction to Information Retrieval

Introduction to Information Retrieval Introduction to Information Retrieval http://informationretrieval.org IIR 5: Index Compression Hinrich Schütze Center for Information and Language Processing, University of Munich 2014-04-17 1/59 Overview

More information

Information Retrieval and Organisation

Information Retrieval and Organisation Information Retrieval and Organisation Dell Zhang Birkbeck, University of London 2016/17 IR Chapter 02 The Term Vocabulary and Postings Lists Constructing Inverted Indexes The major steps in constructing

More information

CS 572: Information Retrieval. Lecture 2: Hello World! (of Text Search)

CS 572: Information Retrieval. Lecture 2: Hello World! (of Text Search) CS 572: Information Retrieval Lecture 2: Hello World! (of Text Search) 1/13/2016 CS 572: Information Retrieval. Spring 2016 1 Course Logistics Lectures: Monday, Wed: 11:30am-12:45pm, W301 Following dates

More information

Ges$one Avanzata dell Informazione Part A Full- Text Informa$on Management. Full- Text Indexing

Ges$one Avanzata dell Informazione Part A Full- Text Informa$on Management. Full- Text Indexing Ges$one Avanzata dell Informazione Part A Full- Text Informa$on Management Full- Text Indexing Contents } Introduction } Inverted Indices } Construction } Searching 2 GAvI - Full- Text Informa$on Management:

More information

Information Retrieval

Information Retrieval Introduction to Information Retrieval Boolean retrieval Basic assumptions of Information Retrieval Collection: Fixed set of documents Goal: Retrieve documents with information that is relevant to the user

More information

Introduction to Information Retrieval IIR 1: Boolean Retrieval

Introduction to Information Retrieval IIR 1: Boolean Retrieval .. Introduction to Information Retrieval IIR 1: Boolean Retrieval Mihai Surdeanu (Based on slides by Hinrich Schütze at informationretrieval.org) Fall 2014 Boolean Retrieval 1 / 77 Take-away Why you should

More information

Lecture 1: Introduction and the Boolean Model

Lecture 1: Introduction and the Boolean Model Lecture 1: Introduction and the Boolean Model Information Retrieval Computer Science Tripos Part II Helen Yannakoudakis 1 Natural Language and Information Processing (NLIP) Group helen.yannakoudakis@cl.cam.ac.uk

More information

Informa(on Retrieval

Informa(on Retrieval Introduc)on to Informa(on Retrieval cs160 Introduction David Kauchak adapted from: h6p://www.stanford.edu/class/cs276/handouts/lecture1 intro.ppt Introduc)ons Name/nickname Dept., college and year One

More information

Information Retrieval CS-E credits

Information Retrieval CS-E credits Information Retrieval CS-E4420 5 credits Tokenization, further indexing issues Antti Ukkonen antti.ukkonen@aalto.fi Slides are based on materials by Tuukka Ruotsalo, Hinrich Schütze and Christina Lioma

More information

CSCI 5417 Information Retrieval Systems! What is Information Retrieval?

CSCI 5417 Information Retrieval Systems! What is Information Retrieval? CSCI 5417 Information Retrieval Systems! Lecture 1 8/23/2011 Introduction 1 What is Information Retrieval? Information retrieval is the science of searching for information in documents, searching for

More information

Information Retrieval

Information Retrieval Introduction to Information Retrieval Introducing Information Retrieval and Web Search Information Retrieval Information Retrieval (IR) is finding material (usually documents) of an unstructurednature

More information

F INTRODUCTION TO WEB SEARCH AND MINING. Kenny Q. Zhu Dept. of Computer Science Shanghai Jiao Tong University

F INTRODUCTION TO WEB SEARCH AND MINING. Kenny Q. Zhu Dept. of Computer Science Shanghai Jiao Tong University F033583 INTRODUCTION TO WEB SEARCH AND MINING Kenny Q. Zhu Dept. of Computer Science Shanghai Jiao Tong University KENNY Q. ZHU Research Interests: Degrees: Postdoc: Experiences: Data & Knowledge Engineering

More information

Introduction to Computational Advertising. MS&E 239 Stanford University Autumn 2010 Instructors: Andrei Broder and Vanja Josifovski

Introduction to Computational Advertising. MS&E 239 Stanford University Autumn 2010 Instructors: Andrei Broder and Vanja Josifovski Introduction to Computational Advertising MS&E 239 Stanford University Autumn 2010 Instructors: Andrei Broder and Vanja Josifovski 1 Lecture 4: Sponsored Search (part 2) 2 Disclaimers This talk presents

More information

Multimedia Information Extraction and Retrieval Term Frequency Inverse Document Frequency

Multimedia Information Extraction and Retrieval Term Frequency Inverse Document Frequency Multimedia Information Extraction and Retrieval Term Frequency Inverse Document Frequency Ralf Moeller Hamburg Univ. of Technology Acknowledgement Slides taken from presentation material for the following

More information

Information Retrieval. (M&S Ch 15)

Information Retrieval. (M&S Ch 15) Information Retrieval (M&S Ch 15) 1 Retrieval Models A retrieval model specifies the details of: Document representation Query representation Retrieval function Determines a notion of relevance. Notion

More information

1Boolean retrieval. information retrieval. term search is quite ambiguous, but in context we use the two synonymously.

1Boolean retrieval. information retrieval. term search is quite ambiguous, but in context we use the two synonymously. 1Boolean retrieval information retrieval The meaning of the term information retrieval (IR) can be very broad. Just getting a credit card out of your wallet so that you can type in the card number is a

More information

PV211: Introduction to Information Retrieval https://www.fi.muni.cz/~sojka/pv211

PV211: Introduction to Information Retrieval https://www.fi.muni.cz/~sojka/pv211 PV211: Introduction to Information Retrieval https://www.fi.muni.cz/~sojka/pv211 IIR 1: Boolean Retrieval Handout version Petr Sojka, Hinrich Schütze et al. Faculty of Informatics, Masaryk University,

More information

Overview of Information Retrieval and Organization. CSC 575 Intelligent Information Retrieval

Overview of Information Retrieval and Organization. CSC 575 Intelligent Information Retrieval Overview of Information Retrieval and Organization CSC 575 Intelligent Information Retrieval 2 How much information? Google: ~100 PB a day; 1+ million servers (est. 15-20 Exabytes stored) Wayback Machine

More information

Querying Introduction to Information Retrieval INF 141 Donald J. Patterson. Content adapted from Hinrich Schütze

Querying Introduction to Information Retrieval INF 141 Donald J. Patterson. Content adapted from Hinrich Schütze Introduction to Information Retrieval INF 141 Donald J. Patterson Content adapted from Hinrich Schütze http://www.informationretrieval.org Overview Boolean Retrieval Weighted Boolean Retrieval Zone Indices

More information

A brief introduction to Information Retrieval

A brief introduction to Information Retrieval 1/64 A brief introduction to Information Retrieval Mark Johnson Department of Computing Macquarie University 2/64 Readings for today s talk Natural Language Processing: Analyzing Text with Python and the

More information

Lecture 1: Introduction and Overview

Lecture 1: Introduction and Overview Lecture 1: Introduction and Overview Information Retrieval Computer Science Tripos Part II Simone Teufel Natural Language and Information Processing (NLIP) Group Simone.Teufel@cl.cam.ac.uk Lent 2014 1

More information

60-538: Information Retrieval

60-538: Information Retrieval 60-538: Information Retrieval September 7, 2017 1 / 48 Outline 1 what is IR 2 3 2 / 48 Outline 1 what is IR 2 3 3 / 48 IR not long time ago 4 / 48 5 / 48 now IR is mostly about search engines there are

More information

Web Information Retrieval Exercises Boolean query answering. Prof. Luca Becchetti

Web Information Retrieval Exercises Boolean query answering. Prof. Luca Becchetti Web Information Retrieval Exercises Boolean query answering Prof. Luca Becchetti Material rif 3. Christopher D. Manning, Prabhakar Raghavan and Hinrich Schueze, Introduction to Information Retrieval, Cambridge

More information

Web Information Retrieval. Lecture 2 Tokenization, Normalization, Speedup, Phrase Queries

Web Information Retrieval. Lecture 2 Tokenization, Normalization, Speedup, Phrase Queries Web Information Retrieval Lecture 2 Tokenization, Normalization, Speedup, Phrase Queries Recap of the previous lecture Basic inverted indexes: Structure: Dictionary and Postings Key step in construction:

More information

Text Retrieval and Web Search IIR 1: Boolean Retrieval

Text Retrieval and Web Search IIR 1: Boolean Retrieval Text Retrieval and Web Search IIR 1: Boolean Retrieval Mihai Surdeanu (Based on slides by Hinrich Schütze at informationretrieval.org) Spring 2017 Boolean Retrieval 1 / 88 Take-away Why you should take

More information

Basic Tokenizing, Indexing, and Implementation of Vector-Space Retrieval

Basic Tokenizing, Indexing, and Implementation of Vector-Space Retrieval Basic Tokenizing, Indexing, and Implementation of Vector-Space Retrieval 1 Naïve Implementation Convert all documents in collection D to tf-idf weighted vectors, d j, for keyword vocabulary V. Convert

More information

n Tuesday office hours changed: n 2-3pm n Homework 1 due Tuesday n Assignment 1 n Due next Friday n Can work with a partner

n Tuesday office hours changed: n 2-3pm n Homework 1 due Tuesday n Assignment 1 n Due next Friday n Can work with a partner Administrative Text Pre-processing and Faster Query Processing" David Kauchak cs458 Fall 2012 adapted from: http://www.stanford.edu/class/cs276/handouts/lecture2-dictionary.ppt Tuesday office hours changed:

More information

Information Retrieval. Lecture 5 - The vector space model. Introduction. Overview. Term weighting. Wintersemester 2007

Information Retrieval. Lecture 5 - The vector space model. Introduction. Overview. Term weighting. Wintersemester 2007 Information Retrieval Lecture 5 - The vector space model Seminar für Sprachwissenschaft International Studies in Computational Linguistics Wintersemester 2007 1/ 28 Introduction Boolean model: all documents

More information

Basic techniques. Text processing; term weighting; vector space model; inverted index; Web Search

Basic techniques. Text processing; term weighting; vector space model; inverted index; Web Search Basic techniques Text processing; term weighting; vector space model; inverted index; Web Search Overview Indexes Query Indexing Ranking Results Application Documents User Information analysis Query processing

More information

Behrang Mohit : txt proc! Review. Bag of word view. Document Named

Behrang Mohit : txt proc! Review. Bag of word view. Document  Named Intro to Text Processing Lecture 9 Behrang Mohit Some ideas and slides in this presenta@on are borrowed from Chris Manning and Dan Jurafsky. Review Bag of word view Document classifica@on Informa@on Extrac@on

More information

Information Retrieval

Information Retrieval Introduction to Information Retrieval Lecture 1: Boolean retrieval 1 Sec. 1.1 Unstructured data in 1680 Which plays of Shakespeare contain the words Brutus AND Caesar but NOT Calpurnia? One could grep

More information

In = number of words appearing exactly n times N = number of words in the collection of words A = a constant. For example, if N=100 and the most

In = number of words appearing exactly n times N = number of words in the collection of words A = a constant. For example, if N=100 and the most In = number of words appearing exactly n times N = number of words in the collection of words A = a constant. For example, if N=100 and the most common word appears 10 times then A = rn*n/n = 1*10/100

More information

Data Modelling and Multimedia Databases M

Data Modelling and Multimedia Databases M ALMA MATER STUDIORUM - UNIERSITÀ DI BOLOGNA Data Modelling and Multimedia Databases M International Second cycle degree programme (LM) in Digital Humanities and Digital Knoledge (DHDK) University of Bologna

More information

Information Retrieval Tutorial 1: Boolean Retrieval

Information Retrieval Tutorial 1: Boolean Retrieval Information Retrieval Tutorial 1: Boolean Retrieval Professor: Michel Schellekens TA: Ang Gao University College Cork 2012-10-26 Boolean Retrieval 1 / 19 Outline 1 Review 2 Boolean Retrieval 2 / 19 Definition

More information

Text Pre-processing and Faster Query Processing

Text Pre-processing and Faster Query Processing Text Pre-processing and Faster Query Processing David Kauchak cs160 Fall 2009 adapted from: http://www.stanford.edu/class/cs276/handouts/lecture2-dictionary.ppt Administrative Everyone have CS lab accounts/access?

More information

Instructor: Stefan Savev

Instructor: Stefan Savev LECTURE 2 What is indexing? Indexing is the process of extracting features (such as word counts) from the documents (in other words: preprocessing the documents). The process ends with putting the information

More information

Preliminary draft (c)2006 Cambridge UP

Preliminary draft (c)2006 Cambridge UP It is a common fallacy, underwritten at this date by the investment of several million dollars in a variety of retrieval hardware, that the algebra of George Boole (1847) is the appropriate formalism for

More information

COSC572 GUEST LECTURE - PROF. GRACE HUI YANG INTRODUCTION TO INFORMATION RETRIEVAL NOV 2, 2016

COSC572 GUEST LECTURE - PROF. GRACE HUI YANG INTRODUCTION TO INFORMATION RETRIEVAL NOV 2, 2016 COSC572 GUEST LECTURE - PROF. GRACE HUI YANG INTRODUCTION TO INFORMATION RETRIEVAL NOV 2, 2016 1 TOPICS FOR TODAY Modes of Search What is Information Retrieval Search vs. Evaluation Vector Space Model

More information

EECS 395/495 Lecture 3 Scalable Indexing, Searching, and Crawling

EECS 395/495 Lecture 3 Scalable Indexing, Searching, and Crawling EECS 395/495 Lecture 3 Scalable Indexing, Searching, and Crawling Doug Downey Based partially on slides by Christopher D. Manning, Prabhakar Raghavan, Hinrich Schütze Announcements Project progress report

More information

Outline of the course

Outline of the course Outline of the course Introduction to Digital Libraries (15%) Description of Information (30%) Access to Information (30%) User Services (10%) Additional topics (15%) Buliding of a (small) digital library

More information

2018 EE448, Big Data Mining, Lecture 8. Search Engines. Weinan Zhang Shanghai Jiao Tong University

2018 EE448, Big Data Mining, Lecture 8. Search Engines. Weinan Zhang Shanghai Jiao Tong University 2018 EE448, Big Data Mining, Lecture 8 Search Engines Weinan Zhang Shanghai Jiao Tong University http://wnzhang.net http://wnzhang.net/teaching/ee448/index.html Acknowledgement and References Referred

More information

CS 6320 Natural Language Processing

CS 6320 Natural Language Processing CS 6320 Natural Language Processing Information Retrieval Yang Liu Slides modified from Ray Mooney s (http://www.cs.utexas.edu/users/mooney/ir-course/slides/) 1 Introduction of IR System components, basic

More information

Efficient Implementation of Postings Lists

Efficient Implementation of Postings Lists Efficient Implementation of Postings Lists Inverted Indices Query Brutus AND Calpurnia J. Pei: Information Retrieval and Web Search -- Efficient Implementation of Postings Lists 2 Skip Pointers J. Pei:

More information

INFO 4300 / CS4300 Information Retrieval. slides adapted from Hinrich Schütze s, linked from

INFO 4300 / CS4300 Information Retrieval. slides adapted from Hinrich Schütze s, linked from INFO 4300 / CS4300 Information Retrieval slides adapted from Hinrich Schütze s, linked from http://informationretrieval.org/ IR 7: Scores in a Complete Search System Paul Ginsparg Cornell University, Ithaca,

More information

Boolean Model. Hongning Wang

Boolean Model. Hongning Wang Boolean Model Hongning Wang CS@UVa Abstraction of search engine architecture Indexed corpus Crawler Ranking procedure Doc Analyzer Doc Representation Query Rep Feedback (Query) Evaluation User Indexer

More information

Data-analysis and Retrieval Boolean retrieval, posting lists and dictionaries

Data-analysis and Retrieval Boolean retrieval, posting lists and dictionaries Data-analysis and Retrieval Boolean retrieval, posting lists and dictionaries Hans Philippi (based on the slides from the Stanford course on IR) April 25, 2018 Boolean retrieval, posting lists & dictionaries

More information

Vector Space Models. Jesse Anderton

Vector Space Models. Jesse Anderton Vector Space Models Jesse Anderton CS6200: Information Retrieval In the first module, we introduced Vector Space Models as an alternative to Boolean Retrieval. This module discusses VSMs in a lot more

More information

Recap: lecture 2 CS276A Information Retrieval

Recap: lecture 2 CS276A Information Retrieval Recap: lecture 2 CS276A Information Retrieval Stemming, tokenization etc. Faster postings merges Phrase queries Lecture 3 This lecture Index compression Space estimation Corpus size for estimates Consider

More information

Chapter III.2: Basic ranking & evaluation measures

Chapter III.2: Basic ranking & evaluation measures Chapter III.2: Basic ranking & evaluation measures 1. TF-IDF and vector space model 1.1. Term frequency counting with TF-IDF 1.2. Documents and queries as vectors 2. Evaluating IR results 2.1. Evaluation

More information

- Content-based Recommendation -

- Content-based Recommendation - - Content-based Recommendation - Institute for Software Technology Inffeldgasse 16b/2 A-8010 Graz Austria 1 Content-based recommendation While CF methods do not require any information about the items,

More information

Recap of the previous lecture. Recall the basic indexing pipeline. Plan for this lecture. Parsing a document. Introduction to Information Retrieval

Recap of the previous lecture. Recall the basic indexing pipeline. Plan for this lecture. Parsing a document. Introduction to Information Retrieval Ch. Introduction to Information Retrieval Recap of the previous lecture Basic inverted indexes: Structure: Dictionary and Postings Lecture 2: The term vocabulary and postings lists Key step in construction:

More information

Information Retrieval. Chap 8. Inverted Files

Information Retrieval. Chap 8. Inverted Files Information Retrieval Chap 8. Inverted Files Issues of Term-Document Matrix 500K x 1M matrix has half-a-trillion 0 s and 1 s Usually, no more than one billion 1 s Matrix is extremely sparse 2 Inverted

More information

Efficiency. Efficiency: Indexing. Indexing. Efficiency Techniques. Inverted Index. Inverted Index (COSC 488)

Efficiency. Efficiency: Indexing. Indexing. Efficiency Techniques. Inverted Index. Inverted Index (COSC 488) Efficiency Efficiency: Indexing (COSC 488) Nazli Goharian nazli@cs.georgetown.edu Difficult to analyze sequential IR algorithms: data and query dependency (query selectivity). O(q(cf max )) -- high estimate-

More information

Boolean retrieval & basics of indexing CE-324: Modern Information Retrieval Sharif University of Technology

Boolean retrieval & basics of indexing CE-324: Modern Information Retrieval Sharif University of Technology Boolean retrieval & basics of indexing CE-324: Modern Information Retrieval Sharif University of Technology M. Soleymani Fall 2014 Most slides have been adapted from: Profs. Manning, Nayak & Raghavan (CS-276,

More information

UNICAL, 21/10/2004. Tutorial goals

UNICAL, 21/10/2004. Tutorial goals Workshop Data Warehousing and Data Mining TEXTEXT MINING INING An Overview of Concepts, Techniques and Applications Ing. Andrea Tagarelli Tutorial goals Introduce you to major aspects of the Knowledge

More information

Relevance of a Document to a Query

Relevance of a Document to a Query Relevance of a Document to a Query Computing the relevance of a document to a query has four parts: 1. Computing the significance of a word within document D. 2. Computing the significance of word to document

More information

modern database systems lecture 4 : information retrieval

modern database systems lecture 4 : information retrieval modern database systems lecture 4 : information retrieval Aristides Gionis Michael Mathioudakis spring 2016 in perspective structured data relational data RDBMS MySQL semi-structured data data-graph representation

More information

Introduction to Information Retrieval

Introduction to Information Retrieval Introduction to Information Retrieval Skiing Seminar Information Retrieval 2010/2011 Introduction to Information Retrieval Prof. Ulrich Müller-Funk, MScIS Andreas Baumgart and Kay Hildebrand Agenda 1 Boolean

More information

A Closeup View. Class Overview CSE 454. Relevance. Retrieval Model Overview. 10/19 IR & Indexing 10/21 Google & Alta.

A Closeup View. Class Overview CSE 454. Relevance. Retrieval Model Overview. 10/19 IR & Indexing 10/21 Google & Alta. Class Overview CSE 454 Infrmation Retrieval & ing Other Cool Stuff Query processing ing IR - Ranking Content Analysis Crawling Network Layer Standard Web Search Engine Architecture 10/19 IR & ing 10/21

More information