VK Multimedia Information Systems

Transcription

1 VK Multimedia Information Systems Mathias Lux, This work is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0

2 Information Retrieval Basics: Agenda Information Retrieval History Information Retrieval & Data Retrieval Searching & Browsing Information Retrieval Models

3 Information Retrieval History Currently there are no museums for IR IR is the process of searching through a document collection based on a particular information need.

4 IR Key Concepts Searching Indexing, Ranking Document Collection Textual, Visual, Auditive Particular Needs Query, User based

5 A History of Libraries Libraries are perfect examples for document collections. Wall paintings in caves e.g. Altamira, ~ 18,500 years old Writing in clay, stone, bones e.g. Mesopotamian cuneiforms, ~ BC e.g. Chinese tortoise-shell carvings, ~ BC e.g. Hieroglyphic inscriptions, Narmer Palette ~ BC

6 A History of Libraries (ctd.) Papyrus Specific plant (subtropical) Organized in rolls, e.g. in Alexandria Parchment Independence from papyrus Sewed together in books Paper Invented in China (bones and bamboo too heavy, silk too expensive) Invention spread -> in 1120 first paper mill in Europe

7 A History of Libraries (ctd.) Gutenberg s printing press (1454) Inexpensive reproduction e.g. Gutenberg Bible Organization & Storage Dewey Decimal System (DDC, 1872) Card Catalog (early 1900s) Microfilm (1930s) MARC (Machine Readable Cataloging, 1960s) Digital computers (1940s+)

8 Library & Archives today Partially converted to electronic catalogues From a certain time point on ( ) Often based on proprietary systems Digitization happens slowly No full text search available Problems with preservation Storage devices & formats

9 History of Searching Browsing Like finding information yourself Catalogs Organized in taxonomies, keywords, etc. Content Based Searching SELECT * FROM books WHERE title= %Search% Information Retrieval ranking, models, weighting link analysis, LSA,...

10 History of IR Starts with development of computers Term Information Retrieval coined by Mooers in 1950 Mooers, C. (March 1950). "The theory of digital handling of nonnumerical information and its implications to machine economics". Proceedings of the meeting of the Association for Computing Machinery at Rutgers University. Two main periods (Spark Jones u. Willett) : academic research models and basics main topics: search & indexing : commercial applications improvement of basic methods

11 A Challenge: The World Wide Web First actual implementation of Hypertext Interconnected documents Linked and referenced World Wide Web (1989, T. Berners-Lee) Unidirectional links (target is not aware) Links are not typed Simple document format & communication protocol (HTML & HTTP) Distributed and not controlled

12 Some IR History Milestones Book Automatic Information Organization and Retrieval, Gerard Salton (1968) Vector space model Paper A statistical interpretation of term specificity and its application in retrieval, Karen Sparck Jones (1972) IDF weighting Book Information Retrieval of C.J. Rijsbergen (1975) Probabilistic model

13 Some IR History Milestones Paper Indexing by Latent Semantic Analysis, S. Deerwester, Susan Dumais, G. W. Furnas, T. K. Landauer, R. Harshman (1990). Latent Semantic Indexing Paper Some simple effective approximations to the 2- Poisson model for probabilistic weighted retrieval Robertsen & Walker (1994) BM25 weighting scheme Paper The Anatomy of a Large-Scale Hypertextual Web Search Engine, Sergey Brin & Larry Page (1998) World wide web retrieval

14 Information Retrieval Basics: Agenda Information Retrieval History Information Retrieval & Data Retrieval Searching & Browsing Information Retrieval Models

15 Organizational: References in the Library Modern Information Retrieval, Ricardo Baeza-Yates & Berthier Ribeiro-Neto, Addison Wesley Google's Pagerank and Beyond: The Science of Search Engine Rankings, Amy N. Langville & Carl D. Meyer, University Presses of CA Readings in Information Retrieval, Karen Sparck Jones, Peter Willett, Morgan Kaufmann

16 Organizational: References On the WWW Skriptum Information Retrieval, Norbert Fuhr, Lecture Notes on Information Retrieval - Univ. Dortmund, Updated in 2002 Information Retrieval 2nd Edt., C.J. Rijsbergen, Butterworth, London 1979 Through me: Lectures on Information Retrieval: Third European Summer-School, Essir 2000 Varenna, Italy, Revised Lectures, Maristella Agosti, Fabio Crestani & Gabriela Pasi (eds.), Lecture Notes in Computer Science, Springer 2000

17 Information Retrieval & Data Retrieval Information Retrieval Data Retrieval Information Level Search Engine Teoma / Google Data Level Data Base Oracle / MySQL

18 Information Retrieval & Data Retrieval Information Retrieval Content Based Search Query ambigous Results ranked by relevance Error tolerant Multiple iterations Examples Search for synonyms Bag of Words Data Retrieval Search for Patterns and String Query formal & unambigous Results not ranked Not error tolerant Clearly defined result set Examples Search for patterns SQL Statement Retrieval is nearly always a combination of both.

19 Information Retrieval Basics: Agenda Information Retrieval History Information Retrieval & Data Retrieval Searching & Browsing Information Retrieval Models

20 Information Retrieval Basics: Searching A user has an information need, which needs to be satisfied. Two different approaches: Browsing Searching

21 Searching & Browsing Searching Explicit information need Definition through query Result lists e.g. Google Browsing User Not necessarily explicit need Navigation through repositories Searching Browsing Docum ents

22 Browsing Flat Browsing User navigates through set of documents No implied ordering, explicit ordering possible Examples: One single directory, one single file Structure Guided Browsing An explicit structure is available for navigation Mostly hierarchical (file directories) Can be generic digraph (WWW) Examples: File systems, World Wide Web

23 Searching Query defines Information Need Ad Hoc Searching Search when you need it Query is created to fit the need Information Filtering Make sets of documents smaller Query is filter criterion Information Push Same as filtering, delivery is different

24 Information Retrieval Basics: Agenda Information Retrieval History Information Retrieval & Data Retrieval Searching & Browsing Information Retrieval Models

25 Information Retrieval System Architecture Aspects Query & languages IR models Documents Internal representation Pre- and post-processing Relevance feedback HCI

26 Information Retrieval Models Boolean Model Set theory & Boolean algebra Vector Model Non binary weights on dimensions Partial match Probabilistic Model Modeling IR in a probabilistic framework

27 Formal Definition of Models An information retrieval model is a quadruple [D, Q, F, R(q i, d j )] D is a set of logical views (or representations) for the documents in the collection. Q is a set of logical views (or representations) for the user needs or queries. F is a framework for modeling document representations, queries and their relationship. R(q i, d j ) is a ranking function which associates a real number with a query q i of Q and a document d j of D.

28 Definitions in Context of Text Retrieval index term word of a document expressing (part of) document semantics weight w i,j quantifies the importance of index term t i for document d j index term vector for document d j (having t different terms in all documents): d ( w, w,..., w ) j 1, j 2, j t, j

29 Boolean Model Based on set theory and Boolean algebra Set of index terms Query is Boolean expression Intuitive concept: Wide usage in bibliographic system Easy implementation and simple formalisms Drawbacks: Binary decision components (true/false) No relevance scale (relevant or not)

30 Boolean Model: Example k a (1,0,0) (1,1,0) k b (1,1,1) k c q k ( k k ) a b c

31 Boolean Model: DNF q k ( k k )... q (1,1,1) (1,1,0) (1,0,0) a b c dnf Express queries in disjunctive normal form (disjunction of conjunctive components) Each of the components is a binary weighted vector associated with (k a,k b,k c ) Weights w i,j {0,1}

32 Boolean Model: Ranking function sim( d, q) j 1 if qcc ( qcc qdnf ) ( k, g ( j ) ( )) i i d gi qcc 0 otherwise similarity is one if one of the conjunctive components in the query is exactly the same as the document term vector.

33 Boolean Model Advantages Clean formalisms Simplicity Disadvantages Might lead to too few / many results No notion of partial match Sequential ordering of terms not taken into account.

34 Vector Model Integrates the notion of partial match Non-binary weights (terms & queries) Degree of similarity computed d ( w, w,..., w ) j 1, j 2, j t, j q ( w, w,..., w ) 1, q 2, q t, q

35 Vector model: Similarity sim( d, q) j d d j j q q t i 1 w w i, j i, q t t 2 2 wi, j wi, q i 1 i 1

36 Vector Model: Example

37 Another Example: Document & Query: D = The quick brown fox jumps over the lazy dog Q = brown lazy fox Results: (1,1,1,1,1,1,1,2) t * (1,1,1,0,0,0,0,0) t = 3 sqrt(11) * sqrt(3) = sqrt(3*11) = sqrt(33) Similarity = 3 / sqrt(33) ~= sim( d, q) j d d j j q q t i 1 w w i, j i, q t t 2 2 wi, j wi, q i 1 i 1

38 Term weighting: TF*IDF Term weighting increases retrieval performance Term frequency How often does a term occur in a document? Most intuitive approach Inverse Document Frequency What is the information content of a term for a document collection? Compare to Information Theory of Shannon

39 Example: IDF 300 documents corpus 3 Term occurs in few documents: High weight for ranking, high discrimination 2,5 2 Term occurs in nearly every document: Low weight for ranking, low discrimination idf 1,5 1 0, Docum ent Frequency

40 Definitions: Normalized Term Frequency f i, j i, j freq i, j max ( freq ) l l, j... normalized term frequency freq... raw term frequency of term i in document j Maximum is computed over all terms in a document Terms which are not present in a document have a raw frequency of 0

41 Definitions: Inverse Document Frequency idf N n i i N log... inverse document frequency for term i n i... number of documents in the corpus... number of document in the corpus which contain term i Note that idf i is independent from the document. Note that the whole corpus has to be taken into account.

42 Why log(...) in IDF? idf value docum ent frequency Logarithm ic IDF No logarithm

43 TF*IDF TF*IDF is a very prominent weighting scheme Works fine, much better than TF or Boolean Quite easy to implement w f log i, j i, j N n i

44 Weighting of query terms w iq, 0.5 fiq, (0.5 ) log max ( f ) N n l l, q i Also using IDF of the corpus But TF is normalized differently TF > 0.5 Note: the query is not part of the corpus!

45 Vector Model Advantages Weighting schemes improve retrieval performance Partial matching allows retrieving documents that approximate query conditions Cosine coefficient allows ranked list output Disadvantages Term are assumed to be mutually independent

46 Simple example (i) Scenario Given a document corpus on birds: nearly each document (say 99%) contains the word bird someone is searching for a document about sparrow nest construction with a query sparrow bird nest construction Exactly the document which would satisfy the user needs does not have the word bird in it.

47 Simple example (ii) TF*IDF weighting knows upon the low discrimative power of the term bird The weight of this term is near to zero This term has virtually no influence on the result list.

48 Exercise 01 Given a document collection... Find the results to a query... Employing the Boolean model Employing the vector model (with TF*IDF) Some hints: Excel: Sheet on homepage Use functions Summenprodukt & Quadratesumme

49 Exercise 01 Document collection (6 documents) spatz, amsel, vogel, drossel, fink, falke, flug spatz, vogel, flug, nest, amsel, amsel, amsel kuckuck, nest, nest, ei, ei, ei, flug, amsel, amsel, vogel amsel, elster, elster, drossel, vogel, ei falke, katze, nest, nest, flug, vogel spatz, spatz, konstruktion, nest, ei Queries: spatz, vogel, nest, konstruktion amsel, ei, nest

50 Exercise 01 d1 d2 d3 d4 d6 d6 idf amsel drossel 1 1 ei elster 2 falke 1 1 fink 1 flug katze 1 konstruktion 1 kuckuck 1 nest spatz vogel ITEC, Klagenfurt University, Austria Multimedia

51 Exercise 02 Create a term vector of a text file with a language of your choice, raw frequency Create a graph

52 Exercise 02 Ideas... Make sure you remove ^[A-Za-z0-9] Make sure to normalize the case ie. by using lower case Print term + \t + value Import output to Excel

53 Don t forget.. To send me the results of Exercise 01 and the graph from Exercise 02

54 Thanks... for your attention!