Outline of the course

Transcription

1 Outline of the course Introduction to Digital Libraries (15%) Description of Information (30%) Access to Information (30%) User Services (10%) Additional topics (15%) Buliding of a (small) digital library Reference material: Ian Witten, David Bainbridge, David Nichols, How to build a Digital Library, Morgan Kaufmann, 2010, ISBN (Second edition) The Web FUB Vittore Casarosa Digital Libraries Part 8-1

2 Where are we Representation of characters within a computer Representation of documents within a computer Text documents Images Audio Video How to store efficiently large amounts of data Compression (of text documents) How to retrieve efficiently the desired item(s) out of large amounts of data Indexing (of text documents) Query execution FUB Vittore Casarosa Digital Libraries Part 8-2

3 Information Retrieval (IR) Information Retrieval (IR) is finding material (usually documents) of an unstructured nature (usually text) that satisfies an information need, from within large collections (usually stored on computers). Research in Information Retrieval started in the seventies, as a field complementary to data base querying (retrieval of structured data) Very often Information Retrieval is also called full text retrieval or free text retrieval Today, the search engines have made free text retrieval the normal way to query for information FUB Vittore Casarosa Digital Libraries Part 8-3

4 Model of Information Retrieval The model of free (or full) text retrieval is: There is a collection of digital documents The user enters a query (usually a few words) The system returns a list of documents ranked in order of relevance to the query In order to do that efficiently: it is necessary first to build an index it is necessary to represent the documents (and the query) in a way suitable for an algorithm to compute the relevance between the query and a document FUB Vittore Casarosa Digital Libraries Part 8-4

5 Indexing In normal life, is the most common way to find content in a book or in a journal For libraries and books, it started a long time ago Catalog (to know where a book is in the library) Table of contents (to know where chapters are in a book) Analytical index (to know where a topic is in a book) Concordances (to know where a word is in a book) Free text retrieval is the extension (by computers) of the concept of concordance in general, not even think of doing a linear scan of the document(s) at the time of the query What is needed is an index of the words (terms) contained in the whole collection FUB Vittore Casarosa Digital Libraries Part 8-5

6 Concordance of search in the Bible FUB Vittore Casarosa Digital Libraries Part 8-6

7 Information in the index A collection is a set of documents, each described (indexed) by a set of representative terms Need to define beforehand what is a document a file, a chapter, a page, a sentence, a word,... Need to define the granularity of the index, i.e. the resolution at which term locations within each document are recorded The index will contain a list of the different terms that appear in the whole collection for each term, the list of documents where the term appears additional term-related related information FUB Vittore Casarosa Digital Libraries Part 8-7

8 Sample collection FUB Vittore Casarosa Digital Libraries Part 8-8

9 Two indexes (inverted files) document frequency lexicon or vocabulary inverted list (postings list) FUB Vittore Casarosa Digital Libraries Part 8-9

10 Processing of input text Obtain character sequence Find enconding (e.g. UTF-8), language, document format, etc Tokenization Apostrophe, p hyphens, compounds, etc. Normalization (equivalence classes) Accents and diacritics (e.g. naive and naïve, resume and résumé) Capitalization, ti case folding C.A.T. CAT cat Stemming organize, organizes, organization organiz Lemmatization am, are, is be car, cars, car s, cars car Stop words a, and, at, be, by, for,... FUB Vittore Casarosa Digital Libraries Part 8-10

11 Tokenization O Neill aren t FUB Vittore Casarosa Digital Libraries Part 8-11

12 Test collections Bible: we all know (each verse is a document) GNUbib: citations to papers in computer science (very short documents) Comact: Commonwealth Acts of Australia (about 1 page documents) TREC: Text Retrieval Conference (documents some very long - from different sources, such as news, US Dept of Energy, Wall Street Journal, etc.) FUB Vittore Casarosa Digital Libraries a term is an alphanumeric sequence up to 256 chars or a 4- digit number Part 8-12

13 Two indexes (inverted files) document frequency lexicon or vocabulary inverted list (postings list) FUB Vittore Casarosa Digital Libraries Part 8-13

14 Index size A document level index needs a value for each pointer (a <term, document> pair) A word-level index needs a value for each word in the collection An uncompressed inverted file can take as much as the text itself For a word level index, assuming that each word appears only once in the documents, we could have 4 bytes for the document pointer; 2 bytes for the word number within the document; result is six bytes of index for each term assuming an average of six bytes per term (in English), the index takes as much space as the text itself; With N documents and f pointers, the minimum number of bits required to identify a document is f x log 2 N For TREC document level: x 20bits = ~ 324 Mbyte (20 is the lowest integer greater than log ) For TREC word level: x 29bits = ~ 1200 Mbyte The use of stop words might give significant savings (about 30%) FUB Vittore Casarosa Digital Libraries Part 8-14

15 Compression of information String of symbols out of a given alphabet encoder uncompressed string network, storage,... uncompressed string decoder compressed string FUB Vittore Casarosa Digital Libraries Part 8-15

16 Compression of text When the textual information is not needed for processing (e.g. when in transit over a network or when stored on secondary storage) it can be represented in a more compact form, i.e. with less bits For text the compression is usually lossless, i.e. the compressed data can be de-compressed to the exact original text Given a string of symbols of a given alphabet (e.g. a string of characters out of the 26 letters of the English alphabet, or a string of numbers out of the 10 digits), which is represented in the computer by N bits, the compression process takes this string and represents it in a different way so thatt after compression the string ti takes n bits, with n<n Compression is not usually noticed (which means that it is well done) but it is used in a number of applications, such as transmission of fax, downloading of web pages, transmission of data over a network, storage of data onto secondary storage, zip files, tar files, etc. FUB Vittore Casarosa Digital Libraries Part 8-16

17 Compression techniques There are two main classes of lossless text compression methods Symbol-wise encoding Dictionary encoding Symbolwise encoding The basic idea is that the most frequent symbols can be coded with less bits (short codewords) than the less frequent symbols (long codewords) Symbol coders work by taking one symbol at the time from the input string, and coding it with a codeword whose length depends on the frequency (probability) of the symbol in the given alphabet The most common symbol encoders are: Huffman coding Arithmetic coding Dictionary coding The basic idea is to replace a sequence of symbols in the input string with an index in a dictionary (list) of phrases Lempel-Ziv 77 Lempel-Ziv 78 Lempel-Ziv-Welch FUB Vittore Casarosa Digital Libraries Part 8-17

18 Symbolwise Morse code (about 1840) FUB Vittore Casarosa Digital Libraries Part 8-18

19 Frequency distribution of the English letters FUB Vittore Casarosa Digital Libraries Part 8-19

20 Index compression Significant savings can be obtained with index compression Each inverted list can be stored as an ascending sequence of integers (document number) For example, the posting list for a term contained in 8 documents might be <8; 3,5,20,21,23,76,77,78> Assuming that the processing of an inverted list is sequential from the beginning, the list can be stored as an initial value followed by d-gaps (the difference between two adjacents document numbers <8; 3,2,15,1,2,53,1,1> In theory, max d-gap is equal to N (number of documents), but on average the size of d-gaps is much smaller than N By estimating the probability distribution of d-gap sizes and by using compression techniques such as Huffman coding, it is possible to represent the list of d-gaps with substantially fewer bits than those needed to represent the list using document numbers FUB Vittore Casarosa Digital Libraries Part 8-20

21 Index compression methods FUB Vittore Casarosa Digital Libraries Part 8-21

22 Storing the lexicon The lexicon is usually stored in the main memory (while the inverted lists are usually stored on disk) The lexicon (the vocabulary) must store the terms, and for each term the address of the inverted file (the postings) Usually also the term frequency (the number of documents containing the term) is stored in the lexicon Other values, usually needed after the inverted list has been retrieved, are stored as part of the inverted lists The lexicon is usually accessed with a binary search or through a hash table Memory requirements depend on the structure of the lexicon FUB Vittore Casarosa Digital Libraries Part 8-22

23 Binary search Example: Find 6 in { 1, 5, 6, 18, 19, 25, 46, 78, 102, 114}. Step 1 (middle element is 19 > 6, take first half): Step 2 (middle element is 9 > 6, take first half): Step 3 (middle element is 5 < 6, take last half): Step 4 (middle element is 6 == 6, done): FUB Vittore Casarosa Digital Libraries Part 8-23

24 Hash tables h a s h f u n c t i o n FUB Vittore Casarosa Digital Libraries Part 8-24

25 Storage requirements for the lexicon storage requirements for one-million-term lexicon FUB Vittore Casarosa Digital Libraries Part 8-25

26 Fixed length strings 20 bytes per term 4 bytes for the document frequency value 4 bytes inverted list address 1 million terms 28MB storage for lexicon FUB Vittore Casarosa Digital Libraries Part 8-26

27 Terminated strings 4 bytes for term frequency 4 bytes for inverted list address 4 bytes for pointer to term 8 bytes (on average) for each term 20MB storage for lexicon FUB Vittore Casarosa Digital Libraries Part 8-27

28 Building the index (sample collection) FUB Vittore Casarosa Digital Libraries Part 8-28

29 Two indexes (inverted files) document frequency lexicon or vocabulary inverted list (postings list) FUB Vittore Casarosa Digital Libraries Part 8-29

30 Building the index (frequency matrix) FUB Vittore Casarosa Digital Libraries Part 8-30

31 Building the index Read the text in document order, one column of the frequency matrix at a time Write the matrix to disk, row by row, in term order (inverted lists) Not possible because of memory requirements Assuming 4 bytes for the term frequency Bible 4 bytes X terms X docs is about 1 GB TREC 4 bytes X X is about 1400 GB Not viable also the use of a large virtual memory, with paging done by the operating system For the Bible, assuming one page fault per pointer, there will be about page faults Assuming 50 page replacement per second, it will take about seconds (about 4 hours) for the Bible, and about two months for TREC FUB Vittore Casarosa Digital Libraries Part 8-31

32 Paged virtual memory virtual memory (on disk) real memory (RAM) slot 1 slot 2 slot 3 page 1 page 2 page 3 page slot 4 slot 5 slot 6 slot slot slot slot 10 slot 11 slot 12 virtual page 2 generates a page fault when referencing virtual page 71 reference to page b virtual page 71 is brought from disk into real memory page 70 page 71 page 72 FUB Vittore Casarosa Digital Libraries Part 8-32

33 Indexer steps: Token sequence Sec. 1.2 Sequence of (Modified token, Document ID) pairs. Doc 1 Doc 2 I did enact Julius Caesar I was killed i' the Capitol; Brutus killed me. So let it be with Caesar. The noble Brutus hath told you Caesar was ambitious FUB Vittore Casarosa Digital Libraries Part 8-33

34 Indexer steps: Sort Sec. 1.2 Sort by terms And then docid Core indexing step FUB Vittore Casarosa Digital Libraries Part 8-34

35 Indexer steps: Dictionary & Postings Sec. 1.2 Multiple term entries in a single document are merged. Split into Dictionary and Postings Doc. frequency information is added. FUB Vittore Casarosa Digital Libraries Part 8-35

36 Linked lists in memory document number term frequency FUB Vittore Casarosa Digital Libraries Part 8-36

37 Typical size and performance figures FUB Vittore Casarosa Digital Libraries Part 8-37

38 Different building methods FUB Vittore Casarosa Digital Libraries Part 8-38

39 Where are we Representation of characters within a computer Representation of documents within a computer Text documents Images Audio Video How to store efficiently large amounts of data Compression (of text documents) How to retrieve efficiently the desired item(s) out of large amounts of data Indexing (of text documents) Query execution FUB Vittore Casarosa Digital Libraries Part 8-39

40 Query execution The execution of a query depend on the underlying data model Structured data Semi-structured data Unstructured data It also depends on whether we want an exact match between the query terms and the documents (boolean query) or a relevance-based retrieval (non boolean) Ranking of the result is important in both cases FUB Vittore Casarosa Digital Libraries Part 8-40

41 Structured data Structured data tends to refer to information in tables Employee Manager Salary Smith Jones Chang Smith Ivy Smith Typically y allows query with numerical range and text exact match (see relational DB and SQL). Salary < AND Manager = Smith. FUB Vittore Casarosa Digital Libraries Part 8-41

42 Semi-structured data In fact almost no data is unstructured E.g., this slide has distinctly identified zones such as the Title and Bullets Facilitates semi-structured search (often called fielded search ) Title contains data AND Bullets contain search Possibility to consider also the linguistic structure (i.e. subject, verb, etc.) FUB Vittore Casarosa Digital Libraries Part 8-42

43 Unstructured data Typically y refers to full text retrieval Allows Keyword (search terms) queries including operators More sophisticated concept queries e.g., find all documents dealing with drug abuse Classic model for searching text documents FUB Vittore Casarosa Digital Libraries Part 8-43

44 Free text queries (non Boolean) The query is a sequence of query terms without (explicit) Boolean connectives Some of the query terms may be missing in a document Not practical to consider the AND of all the query terms Not practical to consider the OR of all the query terms Need to define a method to compute a relevance (or similarity) measure between the query and a document Result will be ranked according to the relevance measure FUB Vittore Casarosa Digital Libraries Part 8-44

45 How to measure relevance The computer (as the name says) can only compute, and therefore we need to define a formula (more in general an algorithm) that takes as input the query and a document, and gives as result a number (i.e. the relevance of the document to the given query) To do that we need to represent the query and each document in some mathematical form, so that they can be fed into the formula In Information Retrieval, given the existence of the lexicon, the most natural way to represent a document in a mathematical form is a vector (a list of numbers), with as many components as there are terms in the lexicon FUB Vittore Casarosa Digital Libraries Part 8-45

46 Simple example FUB Vittore Casarosa Digital Libraries Part 8-46

47 Document as binary vectors lexicon FUB each document (and each query) is represented as a vector (a sequence) of 0 s and 1 s the number of components of the vectors is equal to the size of the lexicon Vittore Casarosa Digital Libraries Part 8-47

48 Similarity measure as inner product A possible (simple) similarity mesasure is the inner product of the two (binary) vectors representing, respectively, the query and a given document X (x 1,x 2,...,x n ) Y(y 1,y 2,,y n ) = x i y i (i=1 to n) Match (hot porridge, D 1 ) = (0,0,0,1,0,0,0,0,1,0) (1,0,0,1,0,0,0,1,1,0) = 2 Three drawbacks of this simple approach of scoring No account of term frequency in the document (i.e. how many times a term appears in the document) No account of term scarcity (in how many documents the term appears) Long documents with many terms are favoured FUB Vittore Casarosa Digital Libraries Part 8-48

49 Document as vectors (of term frequency) FUB each document (and each query) is now represented as a sequence (a vector) of zeros and numbers, i.e. each number is the number of occurrences of the term in the document As before, the number of components of the vectors is equal to the size of the lexicon Vittore Casarosa Digital Libraries Part 8-49

50 Term frequency The first drawback can be solved using the term frequency within the document (i.e. the number of times that the term appears in the document) The binary (terms, documents) matrix becomes now a (term frequency, document) matrix Match (hot porridge, D 1 ) = (0,0,0,1,0,0,0,0,1,0) (1,0,0,1,0,0,0,2,2,0) = 3 FUB Vittore Casarosa Digital Libraries Part 8-50

51 Term weight More generally, we can assign to term t in document d a weight ( ) We can also assign a weight (possibly different) to the same term in the query ( ); usually the weight of terms not in the query is zero The vectors describing the document and the query are now vectors of weights and their inner product gives a similarity measure of the two FUB Vittore Casarosa Digital Libraries Part 8-51

52 Assigning the weight to a term What is usually measured when building the index is the number of times that a term appear in the document (usually called term frequency, or tf) However, it is difficult to assess the relative importance of 0 vs. 1 occurrence of a term in a doc 1 vs. 2 occurrences 2 vs. 3 occurrences While it seems that more is better, it does not necessarily mean that a lot is proportionally better than a few In practice, two common options for a term weight use just the raw tf use a logarithmic formula such as wf 0 if tf t, d 0, 1 logtf t, t, d d otherwise FUB Vittore Casarosa Digital Libraries Part 8-52

53 The logarithmic curve FUB y = log b (x) means x = b y green is base = 10 red is base = e = Vittore Casarosa Digital Libraries purple is base = 1.7 Part 8-53

54 Weighting term scarcity The way to take into account the scarcity (or abundance) of a term in a collection is to count the number of documents in which a term appears (usually called the term document frequency ) and then to consider its inverse (i.e. the inverse document frequency, or idf ) measure of informativeness of a term: its rarity across the whole corpus could just be the inverse of the raw count of number of documents the term occurs in (idf i = 1/df i ) the most commonly used version is: idf i i th b log n df i n is the number of documents in the collection FUB Vittore Casarosa Digital Libraries Part 8-54

55 Final weight: tf x idf (or tf.idf) In conclusion, c o the weight eg of each term i in each document d ( w i,d ) is usually given by the following formula (or very similar variations), called the tf.idf weight w tf log( n / df i, d i, d i tf i, d frequency of term i in document d j n total number of documents df i the number of documents that contain term i Increases with the number of occurrences within a doc Increases with the rarity of the term across the whole corpus ) FUB Vittore Casarosa Digital Libraries Part 8-55

56 Vectors of weights We now have all documents represented as vectors of weights, which take care of both the term frequency (how many times a term appears in a document) and the term document frequency (in how many documents a term appears) We can represent also the query as a vector of weights The simple inner product would still work, but we have not yet taken into account the length of the documents We can therefore change the formula measuring similarity to another one, which does not depend on the length of a document FUB Vittore Casarosa Digital Libraries Part 8-56

57 Similarity in vector space Similarity between vectors d 1 and d 2 captured by the cosine of the angle x between them. Note this is similarity, not distance t 3 d 2 θ d 1 t 1 t 2 FUB Vittore Casarosa Digital Libraries Part 8-57

58 Cosine similarity The cosine of the angle between two vectors can easily computed with the formula X is the modulus (the length) of X The cosine is therefore FUB Vittore Casarosa Digital Libraries Part 8-58

59 Similarity of two documents The cosine formula applied to documents In the formula above the factor Wq can be ignored as it is just a multiplying factor that does not affect the ranking FUB Vittore Casarosa Digital Libraries Part 8-59

60 Summary of retrieval and ranking Build a term-document matrix, assigning a weight to each term in a document (instead of just a binary value as in the simple approach) Usually the weight is tf.idf, i.e. the product of the term frequency (number of occurrences of the term in the document) and the inverse of the term document frequency (number of documents in which the term appears) Consider each document as a vector in n-space (n is the number of distinct terms, i.e. the size of the lexicon) The non-zero components of the vector are the weights of the terms appearing in the document Normalize each vector to unit length (divide each component by the modulus the length of the vector) Consider also the query as a vector in n-space The non-zero components are just the terms appearing in the query (possibly with a weight) Normalize also the query vector Define the similarity measure between the query and a document as the cosine of the angle beteeen the two vectors If both vectors are normalized, the computation is just the inner product of the two vectors FUB Vittore Casarosa Digital Libraries Part 8-60

61 IR topics for next DL Course Queries with a jolly character lab* (easy) *lab and *lab* (doable) lab*r (difficult) Phrase searching Romeo and Juliet Query expansion Use of thesauri User feedback Multilingual search Boosting Index time Query time FUB Vittore Casarosa Digital Libraries Part 8-61

62 Evaluating relevance in IR The evaluation of an IR technique or system is in general done experimentally (rather than analytically). The user has an information need, which is translated into a query Relevance should be assessed relative to the information need and not to the query Information need: I'm looking for information on whether drinking red wine is more effective at reducing your risk of heart attacks than white wine. Query: wine red white heart attack effective Evaluation should be based on whether a doc addresses the information need, not whether it has those words FUB Vittore Casarosa Digital Libraries Part 8-62

63 Relevance measurement FUB Vittore Casarosa Digital Libraries Part 8-63

64 Evaluation Metrics Recall Proportion of retrieved relevant items out of all the relevant items It measures the ability of the system to present all the relevant documents Precision Proportion of retrieved relevant items out of all the retrieved items It measures the ability of the system to present only those items that are relevant FUB Vittore Casarosa Digital Libraries Part 8-64

65 Precision/Recall There will be a high recall (but low precision) by retrieving all docs for all queries Recall is a non-decreasing function of the number of docs retrieved In a good system, precision decreases as either the number of docs retrieved increases or recall increases A fact with strong empirical confirmation Depending on the application, a user may prefer a high precision (e.g. a Google query) or a high recall (e.g. a query to a medical or legal l digital it library) FUB Vittore Casarosa Digital Libraries Part 8-65

66 Where are we Representation of characters within a computer Representation of documents within a computer Text documents Images Audio Video How to store efficiently large amounts of data Compression (of text documents) How to retrieve efficiently the desired item(s) out of large amounts of data Indexing (of text documents) Query execution FUB Vittore Casarosa Digital Libraries Part 8-66