Information Technology for Documentary Data Representation

Transcription

1 ALMA MATER STUDIORUM - UNIVERSITÀ DI BOLOGNA Information Technology for Documentary Data Representation Laurea Magistrale in Scienze del Libro e del Documento University of Bologna Textual Information Retrieval Systems Part I Home page: Electronic version: 1.01.TextualInformationRetrieval-I.pdf I.pdf Electronic version: 1.01.TextualInformationRetrieval-I-2p.pdf Outline Information Retrieval (IR): an introduction Basics on structured and semi-structured data models Textual documents representation in IR systems Automatic indexing techniques Searches of Boolean type Basics on Boolean algebra and truth tables Searches of phrases and for proximity 2

2 Information Retrieval motivation (1) Information Retrieval (IR) deals with the representation, storage, organization, and access to information items The representation and organization of the information items should provide the user with easy access to the information in which she is interested Unfortunately, the characterization of the user information need is not a simple problem!! 3 Information Retrieval motivation (2) Example of user information need in the context of a Cultural Digital Library: Find all documents containing information on Renaissance painters that (1) worked for a Pope and (2) were born outside of Italy. To be relevant the documents must include information about the locations of the artist s most famous paintings and the number artist s painting located in Rome Clearly this full description of user need cannot be used directly to retrieve information using current technologies Instead, the user must translate her information need into a query which can be processed by an IR system Commonly the translation yields a set of keywords (or index terms) which summarize the description of the user information need 4

3 Information Retrieval goal The main task of an IR system is: Given a query, which represents the information needs of the user, and a collection of documents Retrieve the documents in the collection that are relevant to the query, returning them to the user in decreasing order of relevance Document collections are modeled as unstructured data, i.e., data without a schema (default internal organization) able to describe them or to assign a specific semantic 5 Relevant examples of IR applications Web search engines are the most visible IR application (e.g., Bing, Google, Yahoo, etc.) Many universities and public libraries use IR systems to provide access to book, journals, and other documents Among further relevant examples: Social networks applications like Twitter and Facebook; Web digital libraries such as Wikipedia, CiteSeer, Google Scholar, ; Multimedia digital libraries; The Europeana initiative, and the related European Digital Library for the access to cultural digitized objects like books, photos, maps, audio, movies, and archival records from Europe s libraries, archives, museum and audiovisual collections 6

4 Study path We focus on documents representation, their indexing techniques, and models of queries, starting from traditional textual documents and then continuing with the most complex multimedia documents managed by Multimedia Information Retrieval (MIR) systems Before entering into the details of how textual documents are represented by IR systems, we need to provide the minimal set of concepts on structured data and semi-structured data models 7 Let s keep in mind our goal! Facilitate and improve the access to documentary data repositories for general users, conjunctively exploiting: dedicated users manually provided metadata structured data low level features (e.g., document keywords) unstructured data semi-automatically provided annotations semi-structured data Archivio Storico Fiat Cineteca Archivio Artistico Trimotore Fiat G212 Data: 1947 Collezione: Tema di cultura industriale Tipologia: Immagine Aereo, Motore, Ali Das Cabinet des Dr. Caligari Data: 1920 Nazione: Germania Regista: Robert Wiene Genere: Horror Espressionismo, Ipnosi, Sonnambulismo La Gioconda Sito: Museo Louvre, Parigi Secolo: XVI Autore: Leonardo da Vinci Periodo: Rinascimento Data: 1503 Dipinto, Ritratto, Sorriso 8

5 Recall on structured data Structured data base on a predefined schema able to describe the content of the document collection A database (DB) can be seen as a collection of objects representing some information of interest in a structured way (i.e., through a schema) A relational database management systems (RDBMS or just DBMS) is a software system able to manage collections of objects which can be very large (Giga-Tera byte and more) and shared by different applications in a persistent way (even in presence of faults) manage = obtain, elaborate, maintain, produce, distribute Examples of DBMSs: Oracle, IBM (DB2 UDB), Microsoft (SQL Server), Sybase, mysql, PostgreSQL, InterBase 9 Relations as tables DBMSs use the relational model (Codd, 1970) to describe the data, that is the information is organized in tables ( relations ) The rows of table corresponds to records, while the columns correspond to attributes (schema) The language to store/retrieve information from such tables is the Structured Query Language (SQL) Example: if we want to create a table with employees records, so that we can store their employee number, name, age and salary, we can use the following SQL statement: create table EMPLOYEE ( empn integer PRIMARY KEY; name char(50); age integer; salary float ); empn name age salary 10

6 Populating and querying tables Tables can be populated with the SQL insert command, e.g.,: insert into EMPLOYEE values ( 123, Smith, John, 30, ); insert into EMPLOYEE values ( 456, Johnson, Tom, 25, ); empn name age salary We can retrieve information using the select command. E.g., if we want to find all the employees with salary less that 50000, we use the query: Select * From EMPLOYEE Where salary <= Smith, John Johnson, Tom Query execution In absence of access methods (e.g., an index), the DBMS will perform a sequential scanning, checking the salary of each and every employee record against the desired threshold of 50000!!! To accelerate queries execution, we can create an index (usually a B-tree index, as we will see in few minutes) with the command create index E.g., to build an index on the employee s salary, we would issue the SQL statement: create index salidx on EMPLOYEE(salary) In general the DBMS relies on an optimizer component to decide which is the more efficient way to execute a given query sequential vs. index-based evaluation which index is the most appropriate 12

7 Storage hierarchies First level is typically main memory or core or RAM Fast (access time of micro-seconds or faster), small, expensive Second level (secondary store) is typically magnetic disk Much slower (5-10 msec. access time), but much larger and cheaper Database researchers has focused on large databases that do not fit in main memory and thus have to be stored in secondary memory Secondary store is organized into block (= pages) The reason is that, accessing data from the disk involves the mechanical move of the read/write head of the disk above the appropriate track on the disk Because these moves ( seeks ) are slow and expensive, every time we do a disk read we bring into main memory a whole disk block (of the order of 1KB - 8 KB) So, it makes a huge difference of performance if we mange to group similar data in the same disk blocks!! 13 B-tree Access methods, like B-tree, try exactly to achieve good clustering of data in order to minimize the number of disk-reads - Balanced tree of order p - Node: <P1, <K1,Pr1>, P2, <K2, Pr2>,...Pq > q p 5 o 8 o 1 o 3 o 6 o 7 o 9 o 12 o o Pr Data pointer P Tree node pointer Null tree pointer 14

8 B + -tree B-tree variant more commonly used than B-tree - Data pointers only at the leaf nodes - All leaf nodes linked together allows ordered access! Internal node P 1 K 1... K i-1 P i K i... K q-1 P q X X X X K 1 K i < X K i K q-1 < X Leaf node K 1 Pr 1... K i Pr i... K q-1 Pr q-1 P next pointer to next leaf node in tree data pointer data pointer data pointer 15 What else The relational model and SQL provide a large number of additional features, such as: the ability to retrieve information from several tables ( joins ); the matching is based on values! the ability to perform aggregate operations (e.g., sums, averages, etc.) However we restrict the discussion to the above few features which are the essential ones for our purposes 16

9 Recall on semi-structured data Semi-structured data are partially described by means of hierarchical or graph-based models Among relevant models: XML, RDF, OWL, For a complete treatment on the subject, please refer to the course Humanities Computing 17 XML by example XML document example: <Article> <Author> <FirstName>Bob</FirstName> <Surname>Smith</Surname> </Author> <Abstract>This paper concerns... </Abstract> <Section n="1"> <Title>Introduction</Title> <Para>... </Section> </Article> Specific languages (e.g., XQuery, XPath) are used for querying 18

10 XML: from physical too logical representation There is a direct correspondence between the physical representation of an XML document a its logical representation (or document tree) <root> <child> <subchild> </child> <child> </child> </root> </subchild> child subchild root child 19 Back to Information Retrieval models IR deals with unstructured data repositories, i.e., data without a model/schema able to describe them or to assign a specific semantic By focusing on textual data, relevant examples of unstructured (and semi-structured) repositories are: Web (semantic Web); s, social networks, forums, twitter; book digital libraries; scientific document libraries (e.g., CiteSeer); repositories of business documents, patents, etc.; legal documents collections; 20

11 Why unstructured data are that important? On the basis of studies conducted in nineties, users preferred to receive information by other people rather than using an information retrieval system E.g., travel booking The trend has been reversed in the last 10 years thanks to the success of Web technologies and Web search engines E.g., already in 2004 the 92% of the population considered the Web a suitable source for the daily retrieve of useful information Let s keep in mind that: 85% of all stored data is held in unstructured formats 80% of business is conducted on unstructured data Unstructured data double every 3 months 21 Structured vs. unstructured data: in

12 Structured vs. unstructured data: in Textual document representation Textual documents are usually represented as bags (i.e., multi-sets) of representative keywords called index terms following the Bag of Words model Index terms are used to summarize the document content An index term can be: a keyword, chosen from a group of selected words (usually nouns) This approach is particularly useful to classify documents, although it requires a manual intervention any word, also known as full-text indexing Complex index terms may also be defined, such as groups of nouns (e.g., computer science) Alternatively, the composing terms are treated separately and the group is reconstructed by looking at the positions of the words in the text 24

13 Query representation Queries follow a similar approach i.e. a query is a set of words However, how query terms are combined is an issue 25 Boolean queries The simplest retrieval model is based on Boolean algebra: Which plays of Shakespeare contain the words Brutus AND Caesar AND NOT Calpurnia? Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth Antony Brutus Caesar 0 1 Calpurnia Cleopatra mercy 1 0 worser if play contains term, 0 otherwise 26

14 Recall on truth tables A truth table is a mathematical table used in logic to compute the functional values of logical expressions on each of their functional arguments Truth tables can be used to tell whether a propositional expression is true for all legitimate input values, that is, logically valid Among the principle operators: NOT, AND, and OR I. Bartolini Information Technology for Documentary Data Representation 27 Computing the results For each term we have a binary vector, with size N = number of documents in the collection Bit-wise Boolean operations are enough to compute the result: Brutus = (110100), Caesar = (110111), Calpurnia = (010000) (110100) AND (110111) AND NOT (010000) = Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth Antony Brutus Caesar 0 1 Calpurnia Cleopatra mercy 1 0 worser Result =

15 Is the matrix solution a good idea? Assume we have a collection of N = 1M documents Also assume that the overall number of distinct terms is V = 100K, with each document containing, on the average, 1000 distinct terms The matrix consists of 100K x 1M = = 100G boolean values, with only 1% (1G) of 1 s Space overhead suggests to look for a more effective representation Further, consider taking bit-wise AND and OR over vectors of 1M bits The commonest solution adopted in text retrieval system is a structure known as inverted index (also: inverted file ) There are many variants of the inverted index, aiming to: Support different query types Reducing space overhead 29 Building the inverted index (1) doc 1 1) Documents are parsed to extract terms I did enact Julius Caesar I was killed i' the Capitol; Brutus killed me. doc 2 So let it be with Caesar. The noble Brutus hath told you Caesar was ambitious I. Bartolini Term Doc # I 1 did 1 enact 1 julius 1 caesar 1 I 1 was 1 killed 1 i' 1 the 1 capitol 1 brutus 1 killed 1 me 1 so 2 let 2 it 2 be 2 with 2 caesar 2 the 2 noble 2 brutus 2 hath 2 told 2 you 2 caesar 2 was 2 ambitious 2 2) Terms are sorted Term Doc # ambitious 2 be 2 brutus 1 brutus 2 capitol 1 caesar 1 caesar 2 caesar 2 did 1 enact 1 hath 1 I 1 I 1 i' 1 it 2 julius 1 killed 1 killed 1 let 2 me 1 noble 2 so 2 the 1 the 2 told 2 you 2 was 1 was 2 with 2 30

16 Building the inverted index (2) 3) Multiple occurrences of a term 4) The index is then split into a in the same document are dictionary/vocabulary and a merged and frequency posting file Doc # Freq information is added Term Doc # Freq ambitious be brutus brutus capitol caesar caesar 2 2 did enact hath I 1 2 i' it julius killed 1 2 let me noble so the the told you was was with Term N docs Tot Freq ambitious be brutus 2 2 capitol caesar 2 3 did enact hath I 1 2 i' it julius killed 1 2 let me noble so the 2 2 told you was 2 2 with Inverted index size Consider the size of the Dictionary: with 100K terms, even assuming that a vocabulary entry requires 30 bytes on the average, we need just 3MBytes Posting file: if each of the 1M documents contains about 1000 distinct terms, we have 1G entries in the posting file, each of them referenced by a distinct pointer A more effective space utilization is obtained by means of posting lists: For each distinct term, have just one pointer to a list in the posting file This posting list contains the id s of documents for that term and is ordered by increasing values of documents identifiers Continuing with the example, this way we save 1G 100K pointers! Techniques are also available to compress the info within each list Term N docs Tot Freq caesar 2 3 Doc # Freq

17 Using the inverted index with Boolean q. s ANDing two terms is equivalent to intersect their posting lists ORring two terms is equivalent to union their posting lists t1 AND NOT(t2) is equivalent to look for doc id s that are in the posting list of term t1 but not in that of t2 Term N docs Tot Freq computer 5 23 principles 1 3 science 3 20 Doc # Freq q = computer AND science AND principle 5 It is convenient to start processing the shortest lists first, so as to minimize the size of intermediate results We have the Ndocs info in the dictionary! 5 33 What to index? Most common words, like the, a, etc., takes a lot of space since they tend to be present in all the documents At the same time, they provide little or no information at all However, what about searching for to be or not to be? Such words are frequently referred as stopwords A (language-specific) stopword list can be used to filter out those words that are not to be indexed The rule of 30 : ~30 words account for ~30% of all term occurrences in written text Eliminating 150 commonest terms from indexing will cut almost 25% of space Remark: in practice, things are more complex, since we may want to deal with: Punctuation: State-of-the-art, U.S.A. vs. USA, a.out, etc. Numbers: 3/12/9, Mar. 12, 1991, B-52, , etc. 34

18 Stemming In order to save space and to improve the chance of retrieving a document, a process called stemming is usually executed before indexing, so as to reduce terms to their roots e.g., automate(s), automatic, automation all reduced to automat for example compressed and compression are both accepted as equivalent to compress. for exampl compres and compres are both accept as equival to compres. It is experimentally shown that stemming can reduce the number of terms by ~40%, and total index size by ~30% For details on how stemmers (= stemming algorithms) operate: 35 Thesauri Synonyms can be viewed as equivalent terms E.g., car = automobile A text IR system usually comes with a thesaurus that, in its simplest form, consists of: 1. A list of (important) terms 2. For each term, a set of related words Related term = synonyms, hypernyms (car is a kind of), hyponyms ( is a kind of car), etc. Actually, a thesaurus constitutes a semantic network of terms For real examples, please take a look at: Wordnet ( Merriam-Webster thesaurus ( 36

19 The WordNet interface 37 Using thesaurus information If your query includes the term car, there are two possibilities to exploit thesaurus information: 1. The system can expand the query E.g., car AND tyres becomes (car OR automobile) AND tyres 2. The system can build the inverted index by also placing docs containing the term car in the posting list of automobile, and vice versa Usually query expansion is preferred, in order to avoid excessive index growth However, query expansion slows down query processing 38

20 Phrase and proximity queries Searching for phrases (e.g., the challenge of information retrieval ) can be implemented by extending the entries in the posting file with positional information (= position of the term in the document) Term N docs Tot Freq computer 5 23 principles 1 3 science 3 20 q = computer science Doc # Freq 3 2: 13, : 20, 27, 85, 112, : 10 3: 13 2: 5 3: 21, 35, 45 0: 5 2: 28, : 35, 51, Proximity queries are a relaxed version of phrase queries, where we just require that the query terms are close each other E.g., Gates NEAR Microsoft 39 Limits of the Boolean retrieval model Although the Boolean model has a clear semantics and is also appreciated by experienced users, it has several drawbacks: 1. Unexperienced users have difficulties in understanding what Boolean operators really mean 2. No notion of partial matching 3. No ranking of the documents 4. Near-miss and Information overload problems Several extensions of the basic Boolean model have been proposed, in order to solve above problems We skip them, and directly move to consider the vector space model 40