Query-Free News Search

Size: px
Start display at page:

Download "Query-Free News Search"

Transcription

1 Query-Free News Search by Monika Henzinger, Bay-Wei Chang, Sergey Brin - Google Inc. Brian Milch - UC Berkeley presented by Martin Klein, Santosh Vuppala {mklein, svuppala}@cs.odu.edu ODU, Norfolk, 03/21/2007

2 Introduction about news search supporting TV broadcast automatically selecting web pages/news articles relevant to the ongoing stream of TV broadcast news presented to user while watching TV

3 Outline Query Generation (QG) Algorithms Postprocessing Evaluation Conclusion Discussion

4 Examples

5 Examples

6 How is it done? extract queries from closed caption query generation issue queries to news search engine on the web determine what news articles to provide to the user post processing of top results

7 Query Generation relevant articles shown at regular rate during news broadcast ergo, QG needs to produce a query periodically (every S seconds) S = 7 15 (15sec eq ~3 sentences, ~50 words) QG is given the text segment T since the last query generation 7 different QG algorithms to create 2 or 3 word queries

8 TF-IDF Review tf - how often this term appears in text segment T idf - in how many text segments does term appear idf = log(n/f+1) with: N - total number of text segments f - number of text segments in which term appears

9 The Baseline Algorithm A1-BASE simple, calculates the product of tf and idf larger weights for more frequent terms larger weights for more unusual terms returns the two terms with largest weight as the query

10 The A2-IDF2 Algorithm same as baseline but: calculates weight of terms as tf * idf * idf gives even more importance to rare words

11 The Simple Stemming Algorithm A3-STEM assigns a weight to each stem stem of a word is approximated by taking first 5 letters e.g. sports, sportsman, sportive sport for each stem, store all terms (plus their weight) that generated stem c * tf * idf * idf (c = 1 for nouns or else 0.5) top weighted terms for top 2 stems form query

12 The stemming algorithm with compounds A4-COMP A3-STEM + two-word compounds list of allowed compounds (New York, San Francisco) e.g. stem for the compound veterans administration is veter-admin query formed as in A3-STEM can consist of 2, 3 or 4 words now

13 The History Algorithm A5-HIST A4-COMP + history feature uses terms from previous text segments to aid in generating query for the current one data structure called stem vector represents previously seen text (stems, terms, weight) SV combines history information with information (produced by A4-COMP) for current text segment finds the top weighted stems

14 The History Algorithm A5-HIST reset to empty stem vector new SV created for each new T computes similarity score between current T and text in old SV threshold to determine if similar, somewhat similar and dissimilar if similar: SVs are merged if dissimilar: reset old SV and replace with new SV

15 The Query Shortening Algorithm A6-3W identical to A5-HIST issues a multiple term query shortens the query until results are returned from news search engine starts with three terms

16 Algorithm A7-IDF identical to A5-HIST tf * idf replaced by tf * idf * idf

17 Postprocessing issue search queries to news search engine & retrieve top 15 results result contains one news article ~40% duplicates in returned articles near-duplicate backoff strategy detected by comparing titles and summaries if considered as duplicate, dismissed, next in line chosen (if empty, chose 1st)

18 Postprocessing - Boosting use additional high-weighted terms to select most relevant articles (from search results) QG returns: boost terms - top 5 terms (like Q-terms) boost value - their IDF values computes weight for each result re-ranking

19 Postprocessing Similarity Re-ranking similarity between text segment and returned results is computed tf-idf weighted term vectors for text segment and text of article (first 500 chars) compute normalized cosine similarity score can you see the downside?

20 Postprocessing - Filtering discard articles very dissimilar to the caption - F1 if query too vague, top 2 results often dissimilar if found, both articles discarded, unless very similar to caption (cosine similarity score (css)) - F2 F1: if css(t) of page A < threshold b, discard F2: if css of two pages A, B < threshold p, discard BUT if css(t) of A or B > threshold g, retain page (F3)

21 EVALUATION by humans 0 - if the article is not on the topic 1 - if the article is about the topic in general, but not the exact story 2 - if the article is about the exact news story that is being discussed relevant (R) if score of 1 very relevant (R+) if score = 2

22 EVALUATION use Precision and relative Recall to compare algorithms measure topic coverage number of topics with 1 relevant articles

23 Data Sets HN (Headline News): Three 30 minute sessions of CNN headline news, each taken from a different day topics comprise ~ 70min CNN: One hour of Wolf Blitzer Reports on CNN from one day and 30 minutes from another day topics comprise ~ 64min

24 Evaluation of the QG Algorithms

25 Evaluation of the QG Algorithms idf * idf seems to work slightly better than just idf experiments on stemming are inconclusive adding compounds does not improve precision adding history feature is recommended 3-word queries only good without postprocessing

26 Evaluation of Postprocessing filtering gains 20-30% in precision loses 6% in recall filtering + sim. rerank same precision but better recall

27 Query Overlap and URL Overlap reason for similarity in performance of the different query selection algorithms? issue similar queries? return similar URLs?

28 Query Overlap % of queries that have identical terms identical only for similar algorithms

29 URL Overlap % of URLs returned by 2 algorithms high only for similar algorithms

30 Topic Coverage ~70% of the topics have 1 relevant article

31 Filtering Effectiveness % of articles filtered out by what filter rule

32 Filtering Effectiveness how often does a filtering rule make wrong decision?

33 Conclusion 7 algorithms and 3 postprocessing techniques to find news articles on the web top algorithms find relevant article every 16-20sec precision 84%-91% relevant article found in 70% of topics filtering by similarity to caption and with each other improves precision not restricted to news...

34 Background - Authors Brian Milch PhD in CS from UC Berkeley now Post-Doc at MIT, Cambridge, MA Monika Henzinger, Sergey Brin Bay-Wei Chang ( Who Links to Whom:... paper) picture from:

35 Background - Paper published at WWW2003 (Budapest, Hungary) journal article as well (2005) last paper Brin has published (DBLP Bibliography Server)

36 Discussion We have seen that these algorithms work fairly well, they return ~70% of topics with relevant or super relevant articles. Can you think of cases where it would not work? Do you think the algorithms would still achieve ~70% if there were more than 3 options to chose from to evaluate the relevance of the articles? What if they had used a web-scale general topic search engine? Do we know now how to do Query-Free Search? Can we go home?

37 Of Course Not... dimension of the research area We do not have closed captions everywhere! What are movies like Rain Man and Fight Club about? this is how much the paper covers

38

39 References

Query-Free News Search

Query-Free News Search Query-Free News earch Monika Henzinger Google Inc. 2400 Bayshore Parkway Mountain View, CA 94043 UA monika@google.com Brian Milch UC Berkeley Computer cience Division Berkeley, CA 94720-1776 UA milch@cs.berkeley.edu

More information

THE WEB SEARCH ENGINE

THE WEB SEARCH ENGINE International Journal of Computer Science Engineering and Information Technology Research (IJCSEITR) Vol.1, Issue 2 Dec 2011 54-60 TJPRC Pvt. Ltd., THE WEB SEARCH ENGINE Mr.G. HANUMANTHA RAO hanu.abc@gmail.com

More information

Better Contextual Suggestions in ClueWeb12 Using Domain Knowledge Inferred from The Open Web

Better Contextual Suggestions in ClueWeb12 Using Domain Knowledge Inferred from The Open Web Better Contextual Suggestions in ClueWeb12 Using Domain Knowledge Inferred from The Open Web Thaer Samar 1, Alejandro Bellogín 2, and Arjen P. de Vries 1 1 Centrum Wiskunde & Informatica, {samar,arjen}@cwi.nl

More information

Ranking in a Domain Specific Search Engine

Ranking in a Domain Specific Search Engine Ranking in a Domain Specific Search Engine CS6998-03 - NLP for the Web Spring 2008, Final Report Sara Stolbach, ss3067 [at] columbia.edu Abstract A search engine that runs over all domains must give equal

More information

Chapter 6: Information Retrieval and Web Search. An introduction

Chapter 6: Information Retrieval and Web Search. An introduction Chapter 6: Information Retrieval and Web Search An introduction Introduction n Text mining refers to data mining using text documents as data. n Most text mining tasks use Information Retrieval (IR) methods

More information

ΕΠΛ660. Ανάκτηση µε το µοντέλο διανυσµατικού χώρου

ΕΠΛ660. Ανάκτηση µε το µοντέλο διανυσµατικού χώρου Ανάκτηση µε το µοντέλο διανυσµατικού χώρου Σηµερινό ερώτηµα Typically we want to retrieve the top K docs (in the cosine ranking for the query) not totally order all docs in the corpus can we pick off docs

More information

Information Retrieval: Retrieval Models

Information Retrieval: Retrieval Models CS473: Web Information Retrieval & Management CS-473 Web Information Retrieval & Management Information Retrieval: Retrieval Models Luo Si Department of Computer Science Purdue University Retrieval Models

More information

How SPICE Language Modeling Works

How SPICE Language Modeling Works How SPICE Language Modeling Works Abstract Enhancement of the Language Model is a first step towards enhancing the performance of an Automatic Speech Recognition system. This report describes an integrated

More information

CS/INFO 1305 Information Retrieval

CS/INFO 1305 Information Retrieval (Search) Search Using a computer to find relevant pieces of information Text search Idea popularized in the article As We May Think by Vannevar Bush in 1945 Artificial Intelligence Where (or for what)

More information

Home Page. Title Page. Page 1 of 14. Go Back. Full Screen. Close. Quit

Home Page. Title Page. Page 1 of 14. Go Back. Full Screen. Close. Quit Page 1 of 14 Retrieving Information from the Web Database and Information Retrieval (IR) Systems both manage data! The data of an IR system is a collection of documents (or pages) User tasks: Browsing

More information

Relevance of a Document to a Query

Relevance of a Document to a Query Relevance of a Document to a Query Computing the relevance of a document to a query has four parts: 1. Computing the significance of a word within document D. 2. Computing the significance of word to document

More information

Text Analytics (Text Mining)

Text Analytics (Text Mining) CSE 6242 / CX 4242 Apr 1, 2014 Text Analytics (Text Mining) Concepts and Algorithms Duen Horng (Polo) Chau Georgia Tech Some lectures are partly based on materials by Professors Guy Lebanon, Jeffrey Heer,

More information

A NEW CLUSTER MERGING ALGORITHM OF SUFFIX TREE CLUSTERING

A NEW CLUSTER MERGING ALGORITHM OF SUFFIX TREE CLUSTERING A NEW CLUSTER MERGING ALGORITHM OF SUFFIX TREE CLUSTERING Jianhua Wang, Ruixu Li Computer Science Department, Yantai University, Yantai, Shandong, China Abstract: Key words: Document clustering methods

More information

CS473: Course Review CS-473. Luo Si Department of Computer Science Purdue University

CS473: Course Review CS-473. Luo Si Department of Computer Science Purdue University CS473: CS-473 Course Review Luo Si Department of Computer Science Purdue University Basic Concepts of IR: Outline Basic Concepts of Information Retrieval: Task definition of Ad-hoc IR Terminologies and

More information

CS/INFO 1305 Summer 2009

CS/INFO 1305 Summer 2009 Information Retrieval Information Retrieval (Search) IR Search Using a computer to find relevant pieces of information Text search Idea popularized in the article As We May Think by Vannevar Bush in 1945

More information

International Journal of Scientific & Engineering Research Volume 2, Issue 12, December ISSN Web Search Engine

International Journal of Scientific & Engineering Research Volume 2, Issue 12, December ISSN Web Search Engine International Journal of Scientific & Engineering Research Volume 2, Issue 12, December-2011 1 Web Search Engine G.Hanumantha Rao*, G.NarenderΨ, B.Srinivasa Rao+, M.Srilatha* Abstract This paper explains

More information

CLUSTERING, TIERED INDEXES AND TERM PROXIMITY WEIGHTING IN TEXT-BASED RETRIEVAL

CLUSTERING, TIERED INDEXES AND TERM PROXIMITY WEIGHTING IN TEXT-BASED RETRIEVAL STUDIA UNIV. BABEŞ BOLYAI, INFORMATICA, Volume LVII, Number 4, 2012 CLUSTERING, TIERED INDEXES AND TERM PROXIMITY WEIGHTING IN TEXT-BASED RETRIEVAL IOAN BADARINZA AND ADRIAN STERCA Abstract. In this paper

More information

An Information Retrieval Approach for Source Code Plagiarism Detection

An Information Retrieval Approach for Source Code Plagiarism Detection -2014: An Information Retrieval Approach for Source Code Plagiarism Detection Debasis Ganguly, Gareth J. F. Jones CNGL: Centre for Global Intelligent Content School of Computing, Dublin City University

More information

Information Retrieval (IR) Introduction to Information Retrieval. Lecture Overview. Why do we need IR? Basics of an IR system.

Information Retrieval (IR) Introduction to Information Retrieval. Lecture Overview. Why do we need IR? Basics of an IR system. Introduction to Information Retrieval Ethan Phelps-Goodman Some slides taken from http://www.cs.utexas.edu/users/mooney/ir-course/ Information Retrieval (IR) The indexing and retrieval of textual documents.

More information

Text Analytics (Text Mining)

Text Analytics (Text Mining) CSE 6242 / CX 4242 Text Analytics (Text Mining) Concepts and Algorithms Duen Horng (Polo) Chau Georgia Tech Some lectures are partly based on materials by Professors Guy Lebanon, Jeffrey Heer, John Stasko,

More information

CMPSCI 646, Information Retrieval (Fall 2003)

CMPSCI 646, Information Retrieval (Fall 2003) CMPSCI 646, Information Retrieval (Fall 2003) Midterm exam solutions Problem CO (compression) 1. The problem of text classification can be described as follows. Given a set of classes, C = {C i }, where

More information

Basic Tokenizing, Indexing, and Implementation of Vector-Space Retrieval

Basic Tokenizing, Indexing, and Implementation of Vector-Space Retrieval Basic Tokenizing, Indexing, and Implementation of Vector-Space Retrieval 1 Naïve Implementation Convert all documents in collection D to tf-idf weighted vectors, d j, for keyword vocabulary V. Convert

More information

Today s topic CS347. Results list clustering example. Why cluster documents. Clustering documents. Lecture 8 May 7, 2001 Prabhakar Raghavan

Today s topic CS347. Results list clustering example. Why cluster documents. Clustering documents. Lecture 8 May 7, 2001 Prabhakar Raghavan Today s topic CS347 Clustering documents Lecture 8 May 7, 2001 Prabhakar Raghavan Why cluster documents Given a corpus, partition it into groups of related docs Recursively, can induce a tree of topics

More information

Basic techniques. Text processing; term weighting; vector space model; inverted index; Web Search

Basic techniques. Text processing; term weighting; vector space model; inverted index; Web Search Basic techniques Text processing; term weighting; vector space model; inverted index; Web Search Overview Indexes Query Indexing Ranking Results Application Documents User Information analysis Query processing

More information

Designing and Building an Automatic Information Retrieval System for Handling the Arabic Data

Designing and Building an Automatic Information Retrieval System for Handling the Arabic Data American Journal of Applied Sciences (): -, ISSN -99 Science Publications Designing and Building an Automatic Information Retrieval System for Handling the Arabic Data Ibrahiem M.M. El Emary and Ja'far

More information

CC PROCESAMIENTO MASIVO DE DATOS OTOÑO Lecture 7: Information Retrieval II. Aidan Hogan

CC PROCESAMIENTO MASIVO DE DATOS OTOÑO Lecture 7: Information Retrieval II. Aidan Hogan CC5212-1 PROCESAMIENTO MASIVO DE DATOS OTOÑO 2017 Lecture 7: Information Retrieval II Aidan Hogan aidhog@gmail.com How does Google know about the Web? Inverted Index: Example 1 Fruitvale Station is a 2013

More information

An Ensemble Dialogue System for Facts-Based Sentence Generation

An Ensemble Dialogue System for Facts-Based Sentence Generation Track2 Oral Session : Sentence Generation An Ensemble Dialogue System for Facts-Based Sentence Generation Ryota Tanaka, Akihide Ozeki, Shugo Kato, Akinobu Lee Graduate School of Engineering, Nagoya Institute

More information

Efficient query processing

Efficient query processing Efficient query processing Efficient scoring, distributed query processing Web Search 1 Ranking functions In general, document scoring functions are of the form The BM25 function, is one of the best performing:

More information

XML RETRIEVAL. Introduction to Information Retrieval CS 150 Donald J. Patterson

XML RETRIEVAL. Introduction to Information Retrieval CS 150 Donald J. Patterson Introduction to Information Retrieval CS 150 Donald J. Patterson Content adapted from Manning, Raghavan, and Schütze http://www.informationretrieval.org OVERVIEW Introduction Basic XML Concepts Challenges

More information

10/10/13. Traditional database system. Information Retrieval. Information Retrieval. Information retrieval system? Information Retrieval Issues

10/10/13. Traditional database system. Information Retrieval. Information Retrieval. Information retrieval system? Information Retrieval Issues COS 597A: Principles of Database and Information Systems Information Retrieval Traditional database system Large integrated collection of data Uniform access/modifcation mechanisms Model of data organization

More information

Open Source Software Recommendations Using Github

Open Source Software Recommendations Using Github This is the post print version of the article, which has been published in Lecture Notes in Computer Science vol. 11057, 2018. The final publication is available at Springer via https://doi.org/10.1007/978-3-030-00066-0_24

More information

Editor Training Guide

Editor Training Guide Editor Training Guide Getting Started 1. To get into your web site: Open a web browser. Enter your school s web address: If site is not live: Enter temporary web address: Enter temporary username: Enter

More information

Similarity search in multimedia databases

Similarity search in multimedia databases Similarity search in multimedia databases Performance evaluation for similarity calculations in multimedia databases JO TRYTI AND JOHAN CARLSSON Bachelor s Thesis at CSC Supervisor: Michael Minock Examiner:

More information

CS377: Database Systems Text data and information. Li Xiong Department of Mathematics and Computer Science Emory University

CS377: Database Systems Text data and information. Li Xiong Department of Mathematics and Computer Science Emory University CS377: Database Systems Text data and information retrieval Li Xiong Department of Mathematics and Computer Science Emory University Outline Information Retrieval (IR) Concepts Text Preprocessing Inverted

More information

Information Retrieval

Information Retrieval Information Retrieval WS 2016 / 2017 Lecture 2, Tuesday October 25 th, 2016 (Ranking, Evaluation) Prof. Dr. Hannah Bast Chair of Algorithms and Data Structures Department of Computer Science University

More information

Letter Pair Similarity Classification and URL Ranking Based on Feedback Approach

Letter Pair Similarity Classification and URL Ranking Based on Feedback Approach Letter Pair Similarity Classification and URL Ranking Based on Feedback Approach P.T.Shijili 1 P.G Student, Department of CSE, Dr.Nallini Institute of Engineering & Technology, Dharapuram, Tamilnadu, India

More information

CSE 494: Information Retrieval, Mining and Integration on the Internet

CSE 494: Information Retrieval, Mining and Integration on the Internet CSE 494: Information Retrieval, Mining and Integration on the Internet Midterm. 18 th Oct 2011 (Instructor: Subbarao Kambhampati) In-class Duration: Duration of the class 1hr 15min (75min) Total points:

More information

Query Answering Using Inverted Indexes

Query Answering Using Inverted Indexes Query Answering Using Inverted Indexes Inverted Indexes Query Brutus AND Calpurnia J. Pei: Information Retrieval and Web Search -- Query Answering Using Inverted Indexes 2 Document-at-a-time Evaluation

More information

Desktop Crawls. Document Feeds. Document Feeds. Information Retrieval

Desktop Crawls. Document Feeds. Document Feeds. Information Retrieval Information Retrieval INFO 4300 / CS 4300! Web crawlers Retrieving web pages Crawling the web» Desktop crawlers» Document feeds File conversion Storing the documents Removing noise Desktop Crawls! Used

More information

Google Tools and your Library - the Possibilities are Exponential

Google Tools and your Library - the Possibilities are Exponential Google Tools and your Library - the Possibilities are Exponential Google CSE Google Scholar Google Books @ My Library Google Alerts Google Gadgets & igoogle Google Custom Search engine Custom Search Engines

More information

Information Retrieval. CS630 Representing and Accessing Digital Information. What is a Retrieval Model? Basic IR Processes

Information Retrieval. CS630 Representing and Accessing Digital Information. What is a Retrieval Model? Basic IR Processes CS630 Representing and Accessing Digital Information Information Retrieval: Retrieval Models Information Retrieval Basics Data Structures and Access Indexing and Preprocessing Retrieval Models Thorsten

More information

Instructor: Stefan Savev

Instructor: Stefan Savev LECTURE 2 What is indexing? Indexing is the process of extracting features (such as word counts) from the documents (in other words: preprocessing the documents). The process ends with putting the information

More information

IRCE at the NTCIR-12 IMine-2 Task

IRCE at the NTCIR-12 IMine-2 Task IRCE at the NTCIR-12 IMine-2 Task Ximei Song University of Tsukuba songximei@slis.tsukuba.ac.jp Yuka Egusa National Institute for Educational Policy Research yuka@nier.go.jp Masao Takaku University of

More information

SEO. Definitions/Acronyms. Definitions/Acronyms

SEO. Definitions/Acronyms. Definitions/Acronyms Definitions/Acronyms SEO Search Engine Optimization ITS Web Services September 6, 2007 SEO: Search Engine Optimization SEF: Search Engine Friendly SERP: Search Engine Results Page PR (Page Rank): Google

More information

CS246: Mining Massive Datasets Jure Leskovec, Stanford University

CS246: Mining Massive Datasets Jure Leskovec, Stanford University CS246: Mining Massive Datasets Jure Leskovec, Stanford University http://cs246.stanford.edu 2/25/2013 Jure Leskovec, Stanford CS246: Mining Massive Datasets, http://cs246.stanford.edu 3 In many data mining

More information

Notes: Notes: Primo Ranking Customization

Notes: Notes: Primo Ranking Customization Primo Ranking Customization Hello, and welcome to today s lesson entitled Ranking Customization in Primo. Like most search engines, Primo aims to present results in descending order of relevance, with

More information

doi: / _32

doi: / _32 doi: 10.1007/978-3-319-12823-8_32 Simple Document-by-Document Search Tool Fuwatto Search using Web API Masao Takaku 1 and Yuka Egusa 2 1 University of Tsukuba masao@slis.tsukuba.ac.jp 2 National Institute

More information

Using Query History to Prune Query Results

Using Query History to Prune Query Results Using Query History to Prune Query Results Daniel Waegel Ursinus College Department of Computer Science dawaegel@gmail.com April Kontostathis Ursinus College Department of Computer Science akontostathis@ursinus.edu

More information

TREC-10 Web Track Experiments at MSRA

TREC-10 Web Track Experiments at MSRA TREC-10 Web Track Experiments at MSRA Jianfeng Gao*, Guihong Cao #, Hongzhao He #, Min Zhang ##, Jian-Yun Nie**, Stephen Walker*, Stephen Robertson* * Microsoft Research, {jfgao,sw,ser}@microsoft.com **

More information

How Does a Search Engine Work? Part 1

How Does a Search Engine Work? Part 1 How Does a Search Engine Work? Part 1 Dr. Frank McCown Intro to Web Science Harding University This work is licensed under Creative Commons Attribution-NonCommercial 3.0 What we ll examine Web crawling

More information

Implementation of the common phrase index method on the phrase query for information retrieval

Implementation of the common phrase index method on the phrase query for information retrieval Implementation of the common phrase index method on the phrase query for information retrieval Triyah Fatmawati, Badrus Zaman, and Indah Werdiningsih Citation: AIP Conference Proceedings 1867, 020027 (2017);

More information

1. Introduction Accessing imediatrans How to Register as a New User How to Login to imediatrans... 4

1. Introduction Accessing imediatrans How to Register as a New User How to Login to imediatrans... 4 Table of Contents 1. Introduction... 1 2. Accessing imediatrans... 1 2.1 How to Register as a New User... 1 2.2 How to Login to imediatrans... 4 2.2.1. How to Recover the Password... 5 3. Sections in imediatrans...

More information

Level of analysis Finding Out About Chapter 3: 25 Sept 01 R. K. Belew

Level of analysis Finding Out About Chapter 3: 25 Sept 01 R. K. Belew Overview The fascination with the subliminal, the camouflaged, and the encrypted is ancient. Getting a computer to munch away at long strings of letters from the Old Testament is not that different from

More information

Automatic Web Site Authoring with SiteGuide

Automatic Web Site Authoring with SiteGuide Intelligent Information Systems 9999 ISBN 666-666-666, pages 1 14 Automatic Web Site Authoring with SiteGuide V. de Boer, V. Hollink, and M.W. van Someren Human-Computer Studies Laboratory, Informatics

More information

Information Retrieval and Web Search Engines

Information Retrieval and Web Search Engines Information Retrieval and Web Search Engines Lecture 7: Document Clustering May 25, 2011 Wolf-Tilo Balke and Joachim Selke Institut für Informationssysteme Technische Universität Braunschweig Homework

More information

Detecting the Origin of Text Segments Efficiently

Detecting the Origin of Text Segments Efficiently Detecting the Origin of Text Segments Efficiently Ossama Abdel-Hamid Cairo University Giza, Egypt Behshad Behzadi Google Inc Zurich, Switzerland Stefan Christoph Google Inc Zurich, Switzerland o.abdelhamid@fci-cu.edu.eg,

More information

Information Retrieval

Information Retrieval Introduction to Information Retrieval SCCS414: Information Storage and Retrieval Christopher Manning and Prabhakar Raghavan Lecture 10: Text Classification; Vector Space Classification (Rocchio) Relevance

More information

sitecrafting.com

sitecrafting.com sitecrafting.com hello@sitecrafting.com SEARCH ENGINE OPTIMIZATION In its simplest form, Search Engine Optimization is communicating to search engines the intentions of your website so that your website

More information

Outline. Possible solutions. The basic problem. How? How? Relevance Feedback, Query Expansion, and Inputs to Ranking Beyond Similarity

Outline. Possible solutions. The basic problem. How? How? Relevance Feedback, Query Expansion, and Inputs to Ranking Beyond Similarity Outline Relevance Feedback, Query Expansion, and Inputs to Ranking Beyond Similarity Lecture 10 CS 410/510 Information Retrieval on the Internet Query reformulation Sources of relevance for feedback Using

More information

PageRank. CS16: Introduction to Data Structures & Algorithms Spring 2018

PageRank. CS16: Introduction to Data Structures & Algorithms Spring 2018 PageRank CS16: Introduction to Data Structures & Algorithms Spring 2018 Outline Background The Internet World Wide Web Search Engines The PageRank Algorithm Basic PageRank Full PageRank Spectral Analysis

More information

Structural Text Features. Structural Features

Structural Text Features. Structural Features Structural Text Features CISC489/689 010, Lecture #13 Monday, April 6 th Ben CartereGe Structural Features So far we have mainly focused on vanilla features of terms in documents Term frequency, document

More information

Learning Ontology-Based User Profiles: A Semantic Approach to Personalized Web Search

Learning Ontology-Based User Profiles: A Semantic Approach to Personalized Web Search 1 / 33 Learning Ontology-Based User Profiles: A Semantic Approach to Personalized Web Search Bernd Wittefeld Supervisor Markus Löckelt 20. July 2012 2 / 33 Teaser - Google Web History http://www.google.com/history

More information

Chapter 27 Introduction to Information Retrieval and Web Search

Chapter 27 Introduction to Information Retrieval and Web Search Chapter 27 Introduction to Information Retrieval and Web Search Copyright 2011 Pearson Education, Inc. Publishing as Pearson Addison-Wesley Chapter 27 Outline Information Retrieval (IR) Concepts Retrieval

More information

Information Retrieval. (M&S Ch 15)

Information Retrieval. (M&S Ch 15) Information Retrieval (M&S Ch 15) 1 Retrieval Models A retrieval model specifies the details of: Document representation Query representation Retrieval function Determines a notion of relevance. Notion

More information

Information Retrieval

Information Retrieval Information Retrieval Natural Language Processing: Lecture 12 30.11.2017 Kairit Sirts Homework 4 things that seemed to work Bidirectional LSTM instead of unidirectional Change LSTM activation to sigmoid

More information

Introduction & Administrivia

Introduction & Administrivia Introduction & Administrivia Information Retrieval Evangelos Kanoulas ekanoulas@uva.nl Section 1: Unstructured data Sec. 8.1 2 Big Data Growth of global data volume data everywhere! Web data: observation,

More information

Better Contextual Suggestions in ClueWeb12 Using Domain Knowledge Inferred from The Open Web

Better Contextual Suggestions in ClueWeb12 Using Domain Knowledge Inferred from The Open Web Better Contextual Suggestions in ClueWeb12 Using Domain Knowledge Inferred from The Open Web Thaer Samar 1, Alejandro Bellogín 2, and Arjen P. de Vries 1 1 Centrum Wiskunde & Informatica, {samar,arjen}@cwi.nl

More information

How does it work? Typically, a content provider publishes a feed link on its site, which end users subscribe to using a news feed

How does it work? Typically, a content provider publishes a feed link on its site, which end users subscribe to using a news feed ATA Oct07-SF.qxd 10/18/07 7:41 PM Page 32 RSS Feeds: Your Information on Your Terms By Michael Wahlster If you are like most people, you visit several websites regularly. You might seek out professional

More information

Relevance Feedback and Query Reformulation. Lecture 10 CS 510 Information Retrieval on the Internet Thanks to Susan Price. Outline

Relevance Feedback and Query Reformulation. Lecture 10 CS 510 Information Retrieval on the Internet Thanks to Susan Price. Outline Relevance Feedback and Query Reformulation Lecture 10 CS 510 Information Retrieval on the Internet Thanks to Susan Price IR on the Internet, Spring 2010 1 Outline Query reformulation Sources of relevance

More information

M erg in g C lassifiers for Im p ro v ed In fo rm a tio n R e triev a l

M erg in g C lassifiers for Im p ro v ed In fo rm a tio n R e triev a l M erg in g C lassifiers for Im p ro v ed In fo rm a tio n R e triev a l Anette Hulth, Lars Asker Dept, of Computer and Systems Sciences Stockholm University [hulthi asker]ø dsv.su.s e Jussi Karlgren Swedish

More information

Student Guide for Usage of Criterion

Student Guide for Usage of Criterion Student Guide for Usage of Criterion Criterion is an Online Writing Evaluation service offered by ETS. It is a computer-based scoring program designed to help you think about your writing process and communicate

More information

Department of Computer Science and Engineering B.E/B.Tech/M.E/M.Tech : B.E. Regulation: 2013 PG Specialisation : _

Department of Computer Science and Engineering B.E/B.Tech/M.E/M.Tech : B.E. Regulation: 2013 PG Specialisation : _ COURSE DELIVERY PLAN - THEORY Page 1 of 6 Department of Computer Science and Engineering B.E/B.Tech/M.E/M.Tech : B.E. Regulation: 2013 PG Specialisation : _ LP: CS6007 Rev. No: 01 Date: 27/06/2017 Sub.

More information

Semantic Web. Ontology Alignment. Morteza Amini. Sharif University of Technology Fall 95-96

Semantic Web. Ontology Alignment. Morteza Amini. Sharif University of Technology Fall 95-96 ه عا ی Semantic Web Ontology Alignment Morteza Amini Sharif University of Technology Fall 95-96 Outline The Problem of Ontologies Ontology Heterogeneity Ontology Alignment Overall Process Similarity (Matching)

More information

Information Retrieval

Information Retrieval Natural Language Processing SoSe 2014 Information Retrieval Dr. Mariana Neves June 18th, 2014 (based on the slides of Dr. Saeedeh Momtazi) Outline Introduction Indexing Block 2 Document Crawling Text Processing

More information

Lecture 8 May 7, Prabhakar Raghavan

Lecture 8 May 7, Prabhakar Raghavan Lecture 8 May 7, 2001 Prabhakar Raghavan Clustering documents Given a corpus, partition it into groups of related docs Recursively, can induce a tree of topics Given the set of docs from the results of

More information

Search Engine Architecture. Hongning Wang

Search Engine Architecture. Hongning Wang Search Engine Architecture Hongning Wang CS@UVa CS@UVa CS4501: Information Retrieval 2 Document Analyzer Classical search engine architecture The Anatomy of a Large-Scale Hypertextual Web Search Engine

More information

Query Processing and Alternative Search Structures. Indexing common words

Query Processing and Alternative Search Structures. Indexing common words Query Processing and Alternative Search Structures CS 510 Winter 2007 1 Indexing common words What is the indexing overhead for a common term? I.e., does leaving out stopwords help? Consider a word such

More information

Mining Web Data. Lijun Zhang

Mining Web Data. Lijun Zhang Mining Web Data Lijun Zhang zlj@nju.edu.cn http://cs.nju.edu.cn/zlj Outline Introduction Web Crawling and Resource Discovery Search Engine Indexing and Query Processing Ranking Algorithms Recommender Systems

More information

University of Amsterdam at INEX 2010: Ad hoc and Book Tracks

University of Amsterdam at INEX 2010: Ad hoc and Book Tracks University of Amsterdam at INEX 2010: Ad hoc and Book Tracks Jaap Kamps 1,2 and Marijn Koolen 1 1 Archives and Information Studies, Faculty of Humanities, University of Amsterdam 2 ISLA, Faculty of Science,

More information

SEO According to Google

SEO According to Google SEO According to Google An On-Page Optimization Presentation By Rachel Halfhill Lead Copywriter at CDI Agenda Overview Keywords Page Titles URLs Descriptions Heading Tags Anchor Text Alt Text Resources

More information

Welcome to the class of Web Information Retrieval!

Welcome to the class of Web Information Retrieval! Welcome to the class of Web Information Retrieval! Tee Time Topic Augmented Reality and Google Glass By Ali Abbasi Challenges in Web Search Engines Min ZHANG z-m@tsinghua.edu.cn April 13, 2012 Challenges

More information

CS246: Mining Massive Datasets Jure Leskovec, Stanford University

CS246: Mining Massive Datasets Jure Leskovec, Stanford University CS246: Mining Massive Datasets Jure Leskovec, Stanford University http://cs246.stanford.edu 3/6/2012 Jure Leskovec, Stanford CS246: Mining Massive Datasets, http://cs246.stanford.edu 2 In many data mining

More information

Information Retrieval. Lecture 11 - Link analysis

Information Retrieval. Lecture 11 - Link analysis Information Retrieval Lecture 11 - Link analysis Seminar für Sprachwissenschaft International Studies in Computational Linguistics Wintersemester 2007 1/ 35 Introduction Link analysis: using hyperlinks

More information

A Survey on Spoken Document Indexing and Retrieval

A Survey on Spoken Document Indexing and Retrieval A Survey on Spoken Document Indexing and Retrieval Berlin Chen Department of Computer Science & Information Engineering National Taiwan Normal University Introduction (1/5) Ever-increasing volumes of audio-visual

More information

5 Choosing keywords Initially choosing keywords Frequent and rare keywords Evaluating the competition rates of search

5 Choosing keywords Initially choosing keywords Frequent and rare keywords Evaluating the competition rates of search Seo tutorial Seo tutorial Introduction to seo... 4 1. General seo information... 5 1.1 History of search engines... 5 1.2 Common search engine principles... 6 2. Internal ranking factors... 8 2.1 Web page

More information

Chapter 2. Architecture of a Search Engine

Chapter 2. Architecture of a Search Engine Chapter 2 Architecture of a Search Engine Search Engine Architecture A software architecture consists of software components, the interfaces provided by those components and the relationships between them

More information

Duplicate Suppression Improves Google Local Search Ranking

Duplicate Suppression Improves Google Local Search Ranking Duplicate Suppression Improves Google Local Search Ranking 1 Case Study: Duplicate Suppression Improves Search Ranking Introduction For local businesses, SEOs, and brand marketers, Google s local search

More information

Web Search Ranking. (COSC 488) Nazli Goharian Evaluation of Web Search Engines: High Precision Search

Web Search Ranking. (COSC 488) Nazli Goharian Evaluation of Web Search Engines: High Precision Search Web Search Ranking (COSC 488) Nazli Goharian nazli@cs.georgetown.edu 1 Evaluation of Web Search Engines: High Precision Search Traditional IR systems are evaluated based on precision and recall. Web search

More information

CS54701: Information Retrieval

CS54701: Information Retrieval CS54701: Information Retrieval Basic Concepts 19 January 2016 Prof. Chris Clifton 1 Text Representation: Process of Indexing Remove Stopword, Stemming, Phrase Extraction etc Document Parser Extract useful

More information

Data-analysis and Retrieval Boolean retrieval, posting lists and dictionaries

Data-analysis and Retrieval Boolean retrieval, posting lists and dictionaries Data-analysis and Retrieval Boolean retrieval, posting lists and dictionaries Hans Philippi (based on the slides from the Stanford course on IR) April 25, 2018 Boolean retrieval, posting lists & dictionaries

More information

Tilburg University. Authoritative re-ranking of search results Bogers, A.M.; van den Bosch, A. Published in: Advances in Information Retrieval

Tilburg University. Authoritative re-ranking of search results Bogers, A.M.; van den Bosch, A. Published in: Advances in Information Retrieval Tilburg University Authoritative re-ranking of search results Bogers, A.M.; van den Bosch, A. Published in: Advances in Information Retrieval Publication date: 2006 Link to publication Citation for published

More information

Improving Relevance Prediction for Focused Web Crawlers

Improving Relevance Prediction for Focused Web Crawlers 2012 IEEE/ACIS 11th International Conference on Computer and Information Science Improving Relevance Prediction for Focused Web Crawlers Mejdl S. Safran 1,2, Abdullah Althagafi 1 and Dunren Che 1 Department

More information

Exploiting Internal and External Semantics for the Clustering of Short Texts Using World Knowledge

Exploiting Internal and External Semantics for the Clustering of Short Texts Using World Knowledge Exploiting Internal and External Semantics for the Using World Knowledge, 1,2 Nan Sun, 1 Chao Zhang, 1 Tat-Seng Chua 1 1 School of Computing National University of Singapore 2 School of Computer Science

More information

Automatic Summarization

Automatic Summarization Automatic Summarization CS 769 Guest Lecture Andrew B. Goldberg goldberg@cs.wisc.edu Department of Computer Sciences University of Wisconsin, Madison February 22, 2008 Andrew B. Goldberg (CS Dept) Summarization

More information

Applied Databases. Sebastian Maneth. Lecture 11 TFIDF Scoring, Lucene. University of Edinburgh - February 26th, 2017

Applied Databases. Sebastian Maneth. Lecture 11 TFIDF Scoring, Lucene. University of Edinburgh - February 26th, 2017 Applied Databases Lecture 11 TFIDF Scoring, Lucene Sebastian Maneth University of Edinburgh - February 26th, 2017 2 Outline 1. Vector Space Ranking & TFIDF 2. Lucene Next Lecture Assignment 1 marking will

More information

Getting help with Edline 2. Edline basics 3. Displaying a class picture and description 6. Using the News box 7. Using the Calendar box 9

Getting help with Edline 2. Edline basics 3. Displaying a class picture and description 6. Using the News box 7. Using the Calendar box 9 Teacher Guide 1 Henry County Middle School EDLINE March 3, 2003 This guide gives you quick instructions for the most common class-related activities in Edline. Please refer to the online Help for additional

More information

Information Retrieval

Information Retrieval Multimedia Computing: Algorithms, Systems, and Applications: Information Retrieval and Search Engine By Dr. Yu Cao Department of Computer Science The University of Massachusetts Lowell Lowell, MA 01854,

More information

A System for Query-Specific Document Summarization

A System for Query-Specific Document Summarization A System for Query-Specific Document Summarization Ramakrishna Varadarajan, Vagelis Hristidis. FLORIDA INTERNATIONAL UNIVERSITY, School of Computing and Information Sciences, Miami. Roadmap Need for query-specific

More information

Big Data Analytics CSCI 4030

Big Data Analytics CSCI 4030 High dim. data Graph data Infinite data Machine learning Apps Locality sensitive hashing PageRank, SimRank Filtering data streams SVM Recommen der systems Clustering Community Detection Queries on streams

More information

Information Retrieval

Information Retrieval Information Retrieval CSC 375, Fall 2016 An information retrieval system will tend not to be used whenever it is more painful and troublesome for a customer to have information than for him not to have

More information