Outline. Lecture 3: EITN01 Web Intelligence and Information Retrieval. Query languages - aspects. Previous lecture. Anders Ardö.

Transcription

1 Outline Lecture 3: EITN01 Web Intelligence and Information Retrieval Anders Ardö EIT Electrical and Information Technology, Lund University February 5, 2013 A. Ardö, EIT Lecture 3: EITN01 Web Intelligence and Information Retrieval February 5, / 42 A. Ardö, EIT Lecture 3: EITN01 Web Intelligence and Information Retrieval February 5, / 42 Previous lecture Query languages - aspects Query languages/protocols Link based ranking keyword context Boolean natural language pattern matching truncation wild cards regular expressions fuzzy search structured fields A. Ardö, EIT Lecture 3: EITN01 Web Intelligence and Information Retrieval February 5, / 42 A. Ardö, EIT Lecture 3: EITN01 Web Intelligence and Information Retrieval February 5, / 42

2 Query languages Query operations Expressiveness Standardization Examples Structured Query Language - SQL Common Query Language - CQL XQuery Proprietary/home grown Google relevance feedback Give me more like this New Relevance Feedback vector = Original query vector + mean of relevant documents (vectors) in hit set - mean of non-relevant documents (vectors) in hit set query expansion Add or remove search terms Change Boolean operators Change wild cards A. Ardö, EIT Lecture 3: EITN01 Web Intelligence and Information Retrieval February 5, / 42 A. Ardö, EIT Lecture 3: EITN01 Web Intelligence and Information Retrieval February 5, / 42 Network protocols/queries PageRank Dedicated protocol Z39.50 MySQL network protocol PR(u) t = d PR(v) t 1 N v v B u + (1 d)e(u) CGI-script - Web Common Gateway Interface Web services Search/Retrieval via URL - SRU OpenSearch Open Archives Initiative - OAI Random surfer model Click on a random link in the page Eventually gets bored and jumps to a random page Converges to a stable solution Problems size of the Web pages without links - dangling pages (rank sinks) converging link-spamming A. Ardö, EIT Lecture 3: EITN01 Web Intelligence and Information Retrieval February 5, / 42 A. Ardö, EIT Lecture 3: EITN01 Web Intelligence and Information Retrieval February 5, / 42

3 PageRank + TF*IDF Lecture 3 agenda Relevance ranking Chapters 6-7 in Modern Information Retrieval Combine PageRank with vector space model PR(D) sim(q, D) or f (PR(D)) sim(q, D) In practice proximity structure: title, link-anchor text metadata: keywords, description and A. Ardö, EIT Lecture 3: EITN01 Web Intelligence and Information Retrieval February 5, / 42 A. Ardö, EIT Lecture 3: EITN01 Web Intelligence and Information Retrieval February 5, / 42 Outline Introduction - Text main form of communicating knowledge. Document loosely defined, denote a single unit of information. can be any physical unit a file an a Web Page A. Ardö, EIT Lecture 3: EITN01 Web Intelligence and Information Retrieval February 5, / 42 A. Ardö, EIT Lecture 3: EITN01 Web Intelligence and Information Retrieval February 5, / 42

4 Document Metadata Syntax and structure Presentation style - TeX, PDF, HTML, PS, MSWord,... Semantics Information about itself - Metadata Data about the data Descriptive Metadata Author, source, length Dublin Core Metadata Element Set Semantic Metadata Characterizes the subject matter within the document contents A. Ardö, EIT Lecture 3: EITN01 Web Intelligence and Information Retrieval February 5, / 42 A. Ardö, EIT Lecture 3: EITN01 Web Intelligence and Information Retrieval February 5, / 42 Metadata Outline Metadata information on Web documents cataloging, content rating, property rights, digital signatures Standard: RDF - Resource Description Framework description of Web resources to facilitate automated processing of information nodes and attached attribute/values pairs Meta-description of non-textual objects keyword can be used to search the objects A. Ardö, EIT Lecture 3: EITN01 Web Intelligence and Information Retrieval February 5, / 42 A. Ardö, EIT Lecture 3: EITN01 Web Intelligence and Information Retrieval February 5, / 42

5 Markup Languages HTML - example SGML - Standard Generalized Markup Language HTML - HyperText Markup Language XML - extensible Markup Language T E X/ L A T E X <html> <head> <title>test page</title> <meta name="description" content="just my simple testpage"> </head> <body> <h1>hello</h1> It works </body> </html> A. Ardö, EIT Lecture 3: EITN01 Web Intelligence and Information Retrieval February 5, / 42 A. Ardö, EIT Lecture 3: EITN01 Web Intelligence and Information Retrieval February 5, / 42 HTML - example in browser XML - example <?xml version="1.0" encoding="utf-8"?> <bookstore> <book category="xml transforms"> <title lang="en">xslt 1.0</title> <author>evan Lenz</author> <year>2005</year> </book> <book category="xml"> <title lang="en">learning XML</title> <author>erik T. Ray</author> <year>2003</year> <price>39.95</price> </book> </bookstore> A. Ardö, EIT Lecture 3: EITN01 Web Intelligence and Information Retrieval February 5, / 42 A. Ardö, EIT Lecture 3: EITN01 Web Intelligence and Information Retrieval February 5, / 42

6 Text formats Text coding Many formats IR system should retrieve information from different formats Past: IR systems convert the documents Today: IR systems use filters/converters EBCDIC, ASCII, iso-latin Initially, 7 bits. Later, 8 bits characters Unicode character set > characters UTF-8 8 bits for 7bit ASCII, up to 32 bits UTF bits A. Ardö, EIT Lecture 3: EITN01 Web Intelligence and Information Retrieval February 5, / 42 A. Ardö, EIT Lecture 3: EITN01 Web Intelligence and Information Retrieval February 5, / 42 Outline Modeling Natural Language Symbols: separate words or belong to words Symbols are not uniformly distributed Dependency of previous symbols k-order Markovian model We can take words as symbols A. Ardö, EIT Lecture 3: EITN01 Web Intelligence and Information Retrieval February 5, / 42 A. Ardö, EIT Lecture 3: EITN01 Web Intelligence and Information Retrieval February 5, / 42

7 Modeling Natural Language Modeling Natural Language Word distribution inside collections/documents Zipf s law frequency of words decreases as 1 j θ where θ between 1 and 2 Example - word distribution (Zipf s Law) Vocabulary=500, θ = 2 most frequent word: f=300 2nd most frequent: f=76 3rd most frequent: f=33 4th most frequent: f=19 A. Ardö, EIT Lecture 3: EITN01 Web Intelligence and Information Retrieval February 5, / 42 A. Ardö, EIT Lecture 3: EITN01 Web Intelligence and Information Retrieval February 5, / 42 Modeling Natural Language Modeling Natural Language Word distribution inside documents Zipf s law frequency of words decreases as 1 j θ where θ between 1 and 2 Skewed distribution - stopwords Document vocabulary Increases with size (n) of document Heaps law: Vocabulary n β ; β < 1 A. Ardö, EIT Lecture 3: EITN01 Web Intelligence and Information Retrieval February 5, / 42 A. Ardö, EIT Lecture 3: EITN01 Web Intelligence and Information Retrieval February 5, / 42

8 Word (Text): Similarity Models Word (Text): Similarity Models Distance Function Should be symmetric and satisfy triangle inequality Hamming Distance number of positions that have different characters reverse receive Edit (Levenshtein) Distance minimum number of operations needed to make strings equal survey surgery superior for modeling syntatic errors extensions: weights, transpositions, etc A. Ardö, EIT Lecture 3: EITN01 Web Intelligence and Information Retrieval February 5, / 42 A. Ardö, EIT Lecture 3: EITN01 Web Intelligence and Information Retrieval February 5, / 42 Text: Similarity Models Outline Longest Common Subsequence (LCS) survey - surgery LCS: surey Documents: lines as symbols diff in Unix Fingerprints Visual tools A. Ardö, EIT Lecture 3: EITN01 Web Intelligence and Information Retrieval February 5, / 42 A. Ardö, EIT Lecture 3: EITN01 Web Intelligence and Information Retrieval February 5, / 42

9 Representation/Indexing (fig 1.2) Text operations - preprocessing Documents text break into words document numbers words stoplist and *field numbers non-stoplist stemming* Lexical analysis words stemmed term weighting* words * Indicates optional operation terms with weights assign document IDs Index / database 43 1: Lexical analysis 2: Stopwords 3: Stemming 4: Selection of indexing terms 5: Thesaurus A. Ardö, EIT Lecture 3: EITN01 Web Intelligence and Information Retrieval February 5, / 42 A. Ardö, EIT Lecture 3: EITN01 Web Intelligence and Information Retrieval February 5, / 42 Lexical analysis Stopwords convert to words $words=preg_split( /[^a-za-z\ -]+/, $text); numbers punctuation marks hyphens state-of-the-art B-49 case BASIC vs Basic vs basic common words useless for retrieval reduces size to be or not to be A. Ardö, EIT Lecture 3: EITN01 Web Intelligence and Information Retrieval February 5, / 42 A. Ardö, EIT Lecture 3: EITN01 Web Intelligence and Information Retrieval February 5, / 42

10 Stemming Stemming - Porters algorithm removal of affixes connected, connecting, connection, connections connect Porters algorithm Lovins stemmer Paice stemmer police, policy polic lemmatization - full morphological analysis (saw see OR saw?) most common 5 phases rule based fast and reasonably good martin/porterstemmer/ A. Ardö, EIT Lecture 3: EITN01 Web Intelligence and Information Retrieval February 5, / 42 A. Ardö, EIT Lecture 3: EITN01 Web Intelligence and Information Retrieval February 5, / 42 Stemming comparison Term (feature) selection all words noun groups computer science bi-words named entities From Manning, et. al., Introduction to Information Retrieval, A. Ardö, EIT Lecture 3: EITN01 Web Intelligence and Information Retrieval February 5, / 42 A. Ardö, EIT Lecture 3: EITN01 Web Intelligence and Information Retrieval February 5, / 42

11 Thesauri Information theory, text compression list of important words - controlled vocabulary relations between words synonyms useful in the Web context? Take the course EITN45 Information Theory A. Ardö, EIT Lecture 3: EITN01 Web Intelligence and Information Retrieval February 5, / 42 A. Ardö, EIT Lecture 3: EITN01 Web Intelligence and Information Retrieval February 5, / 42