~ Ian Hunneybell: WWWT Revision Notes (15/06/2006) ~

Save this PDF as:
 WORD  PNG  TXT  JPG

Size: px
Start display at page:

Download "~ Ian Hunneybell: WWWT Revision Notes (15/06/2006) ~"

Transcription

1 . Search Engines, history and different types In the beginning there was Archie (990, indexed computer files) and Gopher (99, indexed plain text documents). Lycos (994) and AltaVista (995) were amongst the early popular Search Engines. Google (998) appeared a little later. MSN Search débuted in Google and AltaVista are examples of indexing search engines, which use varying criteria to rank relevant search engine results pages (SERPs). Clusty, a meta-search engine, clusters results based on textual and linguistic similarity. Clusty attempts to provide relevant results more quickly than with a general-purpose search engine, discover relationships between items and view deeper results that would be buried in a purely ranked list. Dogpile, also a meta-search engine, combines the results of several search engines, thus giving potentially greater coverage of the Web, given the assumption that no search engine covers the entire web. Ask Jeeves (now simply Ask), specialises in natural language queries (NLQs). Yahoo! is an example of a directory rather than a search engine, populated by human editors. In the early days of the web this proved an ideal way to navigate the web, but does not scale so well as the web assumes huge proportions. Some search engines have worked on a pay-per-view basis, with companies paying for clickthroughs, or visits to their websites. Overture (originally GoTo.com) is an example of this, and is now part of Yahoo! Search Marketing. A potential extension to this concept is for companies to pay search engines only for completed sales. There would need to be a way to verify sales, and a far higher payment rate could be applied. This is the holy grail of targeted advertising. 2. Challenges faced by Search Engines The web is growing faster than search engines can index (especially directory-based engines such as Yahoo!). Many pages are updated frequently (e.g. News providers) which forces search engines to revisit periodically. Queries are mostly limited to key word searches. Proximity searches offer an improvement, where search terms are limited to same-sentences or paragraphs. Dynamically-generated pages are difficult to index, since they do not persist. The deep web cannot be indexed. Link farms, Link selling (from sites with a high PageRank) and other manipulations attempt to subvert SERPs. A google bomb is a specific attempt to influence the ranking of a web page for a given phrase by adding links to the page with the phrase as the anchor text (e.g. failure ranks the biography of G.W. Bush highest). ~ Page of 7 ~

2 3. How does PageRank work, and how can it be personalised? PR(W) = T/N + (-T)( PR(W )/O(W )+ PR(W 2 )/O(W 2 )+ + PR(W n )/O(W n ) ) where T is the teleportation probability (if T = most of the equation disappears). The PageRank for a given page W is related to the sum of the PageRanks of all the pages linking to W each divided by the number of outlinks for those pages (only a portion of each page s PageRank contributes to W s PageRank). PageRank assigns a user-independent importance to pages. It can be personalised by biasing the algorithm to prefer certain kinds of pages those preferred by the user (e.g. by using features of the URL such as the domain). This will raise the PageRank of pages preferred by the user, thus personalising it. 4. What is the idea behind the random surfer model? PageRank is based on the random surfer model. The model has a parameter 0 a. More precisely, in each time step a surfer, currently visiting a document with probability a follows a link or, with the remaining probability a, randomly jumps to another page. The traditional Random Surfer model does not reflect some very common aspects of Web surfing i.e. reverting to a page previously visited. Reverting to previous pages constitutes a substantial part of Web surfing behaviour and should be reflected in ranking schemes. 5. What is the PageRank rank sink problem and how is it overcome? A rank sink is a loop between web pages, e.g. A B C A which could cause problems when generating PageRank. This problem can be overcome by using a rank source, or the concept of random jumps with a probability. This circumvents the infinite loop scenario. This is described in Page, Brin & Winograd s paper The PageRank Citation Ranking: Bringing Order to the Web (998). 6. Calculating PageRank PageRank is an iterative algorithm which is repeated until the PageRanks stabilise. The teleportation (or random surfer) element has an effect on PageRanks the higher the teleportation probability T, the more the PageRanks tend towards the same value. When T =, all PageRanks are equal. However, when T=0, there is the possibility of a rank sink, which is to be avoided. Page & Brin suggested a value of 0.5 (i.e. 85% of the time a user will traverse by following links). ~ Page 2 of 7 ~

3 (i) calculate adjacency matrix (ii) normalise by number of links (iii) initially all pages have weight = w = (iv) recalculate weights.75 w = 2 Bw = (v) iterate until w k = Bwk 7. What is TF-IDF? Compare PageRank and TF-IDF. Term frequency (TF): A term that appears many times within a document is likely to be more important than a term that appears only once. Inverse Document Frequency (IDF): A term that occurs in a few documents is likely to be a better discriminator that a term that appears in most or all documents. The TF-IDF weight is a statistical measure used to evaluate how important a word is to a document. A high weight in tf idf is reached by a high term frequency (in the given document) and a low document frequency of the term in the whole collection of documents; the weights tend to filter out common terms. e.g. if the word Web appears 8 times in a 200 word document, its TF is 8/200 (0.04). If only 2 documents out of 00 contain the word Web, its IDF is 2/00 (0.02). TD-IDF is therefore 0.04/0.02 = 2. PageRank: The rank of a web page is higher if many pages link to it. Links from highly ranked pages are given greater weight than links from less highly ranked pages. With TF-IDF documents are ranked depending on how well they match a specific query. With PageRank, the pages are ranked in order of importance, with no reference to a specific query. ~ Page 3 of 7 ~

4 8. What is a Markov chain? Why are they useful? A Markov Chain (named after Russian mathematician Andrey Markov), is a network with a probability function imposed on it. It is a statistical model concerning states and transitions with the property that it is memoryless, i.e. it does not remember its previous states (however, real life surfing is not as idealised as this). A Markov Chain has two components: (i) a network structure where each node is called a state; (ii) a transition probability of traversing a link from one state to another. The sum of outgoing probabilities from a state is always, i.e. p = e.g. i i A random walk is a path through the network. The probability of the path is the product of all probabilities along the path (<). We can write a Markov Chain as a sparse matrix, with probabilities in each cell where there is a link. The sum of outgoing probabilities is always. Support s in a chain accepts only paths whose initial probability is above s. Confidence c accepts only paths whose probability is above c. 9. What are Power Law Distributions? Give Examples. Power Laws are those which demonstrate exponential growth and can be seen as straight lines on a log-log graph. k A power law relationship is of the form y = ax which can be expressed as log( y ) = k log( x) + log( a) which itself is of the straight line form y = mx + c. Examples include scale-free networks such as the web (a few highly-connected nodes, many low-connected nodes), Newtonian gravity (inverse-square law), the Pareto principle (wealth distribution aka rule ), earthquake magnitudes (the Richter Scale), storm strengths (the Beaufort scale) and city sizes. Power Laws often have Long Tails the low-frequency, low-amplitude portion of the population. The term was coined in a 2004 article by Chris Anderson in Wired. Hubs & Authorities some nodes are popular (many inlinks, e.g. hubs), others are focal points (many outlinks, e.g. authorities). Many are both. Preferential Attachment aka the rich get richer. Popular sites tend to become more popular, thus accentuating the power law distribution. ~ Page 4 of 7 ~

5 0. The Bow-Tie theory This model suggests that only a portion of the Web (the core) is well-connected. There are many pages which link into the core, but are not linked to (e.g. new pages) and many which are linked to but do not link back to the core (e.g. corporate websites). There are also tubes and tendrils which avoid the core, and disconnected pages.. What are recall/precision in the context of Search Engines? Recall is the ability of a search engine to obtain all of the relevant documents. Precision is the ratio or fraction of a search output that is relevant for a particular query. There is a certain amount of subjectivity with these terms, and it is generally not possible to know the total number of relevant documents in the case of calculating recall. 2. Recommender Systems and Collaborative Filtering. What is collaborative filtering? What is the difference between user-based and item-based collaborative filtering? Recommender systems are programs which make predictions, often implemented using collaborative filtering algorithms. Data can be gathered implicitly (e.g. recording purchases or page impressions) or explicitly (asking for ratings e.g. IMDB or relative rankings). Collaborative filtering (CF) is the method of making automatic predictions (filtering) about the interests of a user by collecting taste information from many users (collaborating). The underlying assumption of CF approach is that those who agreed in the past tend to agree again in the future. User-based collaborative filtering predicts a user s interests based on information from similar user profiles. A user-item matrix is constructed, and correlation between nearneighbours (users) can be calculated (treating the user information as vectors). Item-based collaborative filtering (e.g. Amazon.com, users who bought x also bought y ) works by building an item-item matrix which determines relationships between pairs of items, and using the matrix and the data on the current user, makes inferences. Difficulties with CF include sparsity (many users, but few ratings), scalability (huge numbers of users and items, e.g. customers and books) and the first-rater problem (i.e cold-starting). ~ Page 5 of 7 ~

6 3. Web Data Mining contrast content mining, link mining and usage mining. Content mining concerns mining useful information or knowledge from web page contents. Usage mining refers to the discovery of user access patterns from Web usage logs. Web Link mining tries to discover useful knowledge from the structure of hyperlinks. Content mining retrieves and analyses web content, classifying web objects (pages, websites, etc). Objects can be clustered and web crawlers can use a focused search to reveal patterns. DMOZ (the Open Directory Project) is an example of clustering by classification. Applications of usage mining include website reorganisation, caching oft-accessed pages, recommendation and personalisation. Users can be identified by IP address (unreliable), cookies (security/deleted cookie issues), logins. Sessions can be identified by time-stamping or by navigation connections. Sessions illustrate user trails through a website. Markov chains can be constructed from user trails. By adding start and end states, the chains are termed absorbing (i.e. selfcontained). Pattern discovery tools can identify complex patterns in usage. PageRank and Hubs and Authorities are the two most well-known applications of link mining. Other forms of mining include association rule mining, market basket analysis and cluster analysis. 4. What is capture/recapture? This is a simple mechanism for estimating the size of a population. We begin by capturing a sample of size c, tagging them, and then returning them to the pond. After allowing some time for mixing, a second recaptured sample of size r is taken and the number t of them which are tagged is recorded. An estimate of the number of fish in the pond is c r N = t Put another way, the ratio of recaptured to captured (t/c) is estimated to be the same as the ratio of recaptured sample size to total population (r/n). In the context of the size of the web, we can use two search engines results to simulate the capture/recapture, with the overlap providing the recapture value. We need to know the size of the web as reported by one of the search engines. The equation thus becomes size of SE # pages returned by SE2 Size of web = overlap between SE& SE2 This is becoming a less precise metric as more of the web consists of dynamic pages. It is no longer possible to determine the size of the web in terms of pages. ~ Page 6 of 7 ~

7 5. RSS (Really Simple Syndication) RSS is a way to publish and receive information, without using . Advantages are that recipients opt-in to receiving an RSS feed, it is not filtered (like ) and can be updated as often as required. RSS feeds can be read using an RSS Feed Aggregator. RSS is pulled by a recipient (or its feed aggregator), not pushed by the publisher. RSS is based on XML. The RSS specification describes the structure of the XML necessary to deliver RSS information. A minimal structure is <rss version="2.0"> <channel> <title>rss feed title</title> <link>http://www.myfeed.com/</link> <description>short description of feed</description> <item> <title>story headline</title> <description>summary of story</description> <link>http://www.myfeed.com/story</link> </item> </channel> </rss> Channels can also include an image, published date, editor, etc. There can be many items within a channel. Items can also include author, category, published date, etc. ~ Page 7 of 7 ~

Mining Web Data. Lijun Zhang

Mining Web Data. Lijun Zhang Mining Web Data Lijun Zhang zlj@nju.edu.cn http://cs.nju.edu.cn/zlj Outline Introduction Web Crawling and Resource Discovery Search Engine Indexing and Query Processing Ranking Algorithms Recommender Systems

More information

Mining Web Data. Lijun Zhang

Mining Web Data. Lijun Zhang Mining Web Data Lijun Zhang zlj@nju.edu.cn http://cs.nju.edu.cn/zlj Outline Introduction Web Crawling and Resource Discovery Search Engine Indexing and Query Processing Ranking Algorithms Recommender Systems

More information

Information Retrieval Lecture 4: Web Search. Challenges of Web Search 2. Natural Language and Information Processing (NLIP) Group

Information Retrieval Lecture 4: Web Search. Challenges of Web Search 2. Natural Language and Information Processing (NLIP) Group Information Retrieval Lecture 4: Web Search Computer Science Tripos Part II Simone Teufel Natural Language and Information Processing (NLIP) Group sht25@cl.cam.ac.uk (Lecture Notes after Stephen Clark)

More information

COMP Page Rank

COMP Page Rank COMP 4601 Page Rank 1 Motivation Remember, we were interested in giving back the most relevant documents to a user. Importance is measured by reference as well as content. Think of this like academic paper

More information

Information Retrieval (IR) Introduction to Information Retrieval. Lecture Overview. Why do we need IR? Basics of an IR system.

Information Retrieval (IR) Introduction to Information Retrieval. Lecture Overview. Why do we need IR? Basics of an IR system. Introduction to Information Retrieval Ethan Phelps-Goodman Some slides taken from http://www.cs.utexas.edu/users/mooney/ir-course/ Information Retrieval (IR) The indexing and retrieval of textual documents.

More information

A Survey on Web Information Retrieval Technologies

A Survey on Web Information Retrieval Technologies A Survey on Web Information Retrieval Technologies Lan Huang Computer Science Department State University of New York, Stony Brook Presented by Kajal Miyan Michigan State University Overview Web Information

More information

Lec 8: Adaptive Information Retrieval 2

Lec 8: Adaptive Information Retrieval 2 Lec 8: Adaptive Information Retrieval 2 Advaith Siddharthan Introduction to Information Retrieval by Manning, Raghavan & Schütze. Website: http://nlp.stanford.edu/ir-book/ Linear Algebra Revision Vectors:

More information

Lecture 11: Graph algorithms! Claudia Hauff (Web Information Systems)!

Lecture 11: Graph algorithms! Claudia Hauff (Web Information Systems)! Lecture 11: Graph algorithms!! Claudia Hauff (Web Information Systems)! ti2736b-ewi@tudelft.nl 1 Course content Introduction Data streams 1 & 2 The MapReduce paradigm Looking behind the scenes of MapReduce:

More information

Part 1: Link Analysis & Page Rank

Part 1: Link Analysis & Page Rank Chapter 8: Graph Data Part 1: Link Analysis & Page Rank Based on Leskovec, Rajaraman, Ullman 214: Mining of Massive Datasets 1 Graph Data: Social Networks [Source: 4-degrees of separation, Backstrom-Boldi-Rosa-Ugander-Vigna,

More information

Information Retrieval May 15. Web retrieval

Information Retrieval May 15. Web retrieval Information Retrieval May 15 Web retrieval What s so special about the Web? The Web Large Changing fast Public - No control over editing or contents Spam and Advertisement How big is the Web? Practically

More information

CS224W: Social and Information Network Analysis Jure Leskovec, Stanford University

CS224W: Social and Information Network Analysis Jure Leskovec, Stanford University CS224W: Social and Information Network Analysis Jure Leskovec, Stanford University http://cs224w.stanford.edu How to organize the Web? First try: Human curated Web directories Yahoo, DMOZ, LookSmart Second

More information

Link Analysis in Web Mining

Link Analysis in Web Mining Problem formulation (998) Link Analysis in Web Mining Hubs and Authorities Spam Detection Suppose we are given a collection of documents on some broad topic e.g., stanford, evolution, iraq perhaps obtained

More information

Introduction to Information Retrieval

Introduction to Information Retrieval Introduction to Information Retrieval http://informationretrieval.org IIR 21: Link Analysis Hinrich Schütze Center for Information and Language Processing, University of Munich 2014-06-18 1/80 Overview

More information

Today we show how a search engine works

Today we show how a search engine works How Search Engines Work Today we show how a search engine works What happens when a searcher enters keywords What was performed well in advance Also explain (briefly) how paid results are chosen If we

More information

Web Structure Mining using Link Analysis Algorithms

Web Structure Mining using Link Analysis Algorithms Web Structure Mining using Link Analysis Algorithms Ronak Jain Aditya Chavan Sindhu Nair Assistant Professor Abstract- The World Wide Web is a huge repository of data which includes audio, text and video.

More information

How to organize the Web?

How to organize the Web? How to organize the Web? First try: Human curated Web directories Yahoo, DMOZ, LookSmart Second try: Web Search Information Retrieval attempts to find relevant docs in a small and trusted set Newspaper

More information

CS224W: Social and Information Network Analysis Jure Leskovec, Stanford University

CS224W: Social and Information Network Analysis Jure Leskovec, Stanford University CS224W: Social and Information Network Analysis Jure Leskovec, Stanford University http://cs224w.stanford.edu How to organize the Web? First try: Human curated Web directories Yahoo, DMOZ, LookSmart Second

More information

Big Data Analytics CSCI 4030

Big Data Analytics CSCI 4030 High dim. data Graph data Infinite data Machine learning Apps Locality sensitive hashing PageRank, SimRank Filtering data streams SVM Recommen der systems Clustering Community Detection Web advertising

More information

Lecture Notes to Big Data Management and Analytics Winter Term 2017/2018 Node Importance and Neighborhoods

Lecture Notes to Big Data Management and Analytics Winter Term 2017/2018 Node Importance and Neighborhoods Lecture Notes to Big Data Management and Analytics Winter Term 2017/2018 Node Importance and Neighborhoods Matthias Schubert, Matthias Renz, Felix Borutta, Evgeniy Faerman, Christian Frey, Klaus Arthur

More information

Information Retrieval Spring Web retrieval

Information Retrieval Spring Web retrieval Information Retrieval Spring 2016 Web retrieval The Web Large Changing fast Public - No control over editing or contents Spam and Advertisement How big is the Web? Practically infinite due to the dynamic

More information

Agenda. Math Google PageRank algorithm. 2 Developing a formula for ranking web pages. 3 Interpretation. 4 Computing the score of each page

Agenda. Math Google PageRank algorithm. 2 Developing a formula for ranking web pages. 3 Interpretation. 4 Computing the score of each page Agenda Math 104 1 Google PageRank algorithm 2 Developing a formula for ranking web pages 3 Interpretation 4 Computing the score of each page Google: background Mid nineties: many search engines often times

More information

Searching the Web [Arasu 01]

Searching the Web [Arasu 01] Searching the Web [Arasu 01] Most user simply browse the web Google, Yahoo, Lycos, Ask Others do more specialized searches web search engines submit queries by specifying lists of keywords receive web

More information

Big Data Analytics CSCI 4030

Big Data Analytics CSCI 4030 High dim. data Graph data Infinite data Machine learning Apps Locality sensitive hashing PageRank, SimRank Filtering data streams SVM Recommen der systems Clustering Community Detection Web advertising

More information

Search Engines. Information Retrieval in Practice

Search Engines. Information Retrieval in Practice Search Engines Information Retrieval in Practice All slides Addison Wesley, 2008 Web Crawler Finds and downloads web pages automatically provides the collection for searching Web is huge and constantly

More information

A Modified Algorithm to Handle Dangling Pages using Hypothetical Node

A Modified Algorithm to Handle Dangling Pages using Hypothetical Node A Modified Algorithm to Handle Dangling Pages using Hypothetical Node Shipra Srivastava Student Department of Computer Science & Engineering Thapar University, Patiala, 147001 (India) Rinkle Rani Aggrawal

More information

Unit VIII. Chapter 9. Link Analysis

Unit VIII. Chapter 9. Link Analysis Unit VIII Link Analysis: Page Ranking in web search engines, Efficient Computation of Page Rank using Map-Reduce and other approaches, Topic-Sensitive Page Rank, Link Spam, Hubs and Authorities (Text Book:2

More information

Information Retrieval

Information Retrieval Information Retrieval Additional Reference Introduction to Information Retrieval, Manning, Raghavan and Schütze, online book: http://nlp.stanford.edu/ir-book/ Why Study Information Retrieval? Google Searches

More information

INTRODUCTION TO DATA SCIENCE. Link Analysis (MMDS5)

INTRODUCTION TO DATA SCIENCE. Link Analysis (MMDS5) INTRODUCTION TO DATA SCIENCE Link Analysis (MMDS5) Introduction Motivation: accurate web search Spammers: want you to land on their pages Google s PageRank and variants TrustRank Hubs and Authorities (HITS)

More information

Introduction to Data Mining

Introduction to Data Mining Introduction to Data Mining Lecture #10: Link Analysis-2 Seoul National University 1 In This Lecture Pagerank: Google formulation Make the solution to converge Computing Pagerank for very large graphs

More information

Part I: Data Mining Foundations

Part I: Data Mining Foundations Table of Contents 1. Introduction 1 1.1. What is the World Wide Web? 1 1.2. A Brief History of the Web and the Internet 2 1.3. Web Data Mining 4 1.3.1. What is Data Mining? 6 1.3.2. What is Web Mining?

More information

Single link clustering: 11/7: Lecture 18. Clustering Heuristics 1

Single link clustering: 11/7: Lecture 18. Clustering Heuristics 1 Graphs and Networks Page /7: Lecture 8. Clustering Heuristics Wednesday, November 8, 26 8:49 AM Today we will talk about clustering and partitioning in graphs, and sometimes in data sets. Partitioning

More information

Information Retrieval and Web Search Engines

Information Retrieval and Web Search Engines Information Retrieval and Web Search Engines Lecture 12: Link Analysis January 28 th, 2016 Wolf-Tilo Balke and Younes Ghammad Institut für Informationssysteme Technische Universität Braunschweig An Overview

More information

Graph and Web Mining - Motivation, Applications and Algorithms PROF. EHUD GUDES DEPARTMENT OF COMPUTER SCIENCE BEN-GURION UNIVERSITY, ISRAEL

Graph and Web Mining - Motivation, Applications and Algorithms PROF. EHUD GUDES DEPARTMENT OF COMPUTER SCIENCE BEN-GURION UNIVERSITY, ISRAEL Graph and Web Mining - Motivation, Applications and Algorithms PROF. EHUD GUDES DEPARTMENT OF COMPUTER SCIENCE BEN-GURION UNIVERSITY, ISRAEL Web mining - Outline Introduction Web Content Mining Web usage

More information

Using PageRank in Feature Selection

Using PageRank in Feature Selection Using PageRank in Feature Selection Dino Ienco, Rosa Meo, and Marco Botta Dipartimento di Informatica, Università di Torino, Italy {ienco,meo,botta}@di.unito.it Abstract. Feature selection is an important

More information

COMP 4601 Hubs and Authorities

COMP 4601 Hubs and Authorities COMP 4601 Hubs and Authorities 1 Motivation PageRank gives a way to compute the value of a page given its position and connectivity w.r.t. the rest of the Web. Is it the only algorithm: No! It s just one

More information

Using PageRank in Feature Selection

Using PageRank in Feature Selection Using PageRank in Feature Selection Dino Ienco, Rosa Meo, and Marco Botta Dipartimento di Informatica, Università di Torino, Italy fienco,meo,bottag@di.unito.it Abstract. Feature selection is an important

More information

Today we shall be starting discussion on search engines and web crawler.

Today we shall be starting discussion on search engines and web crawler. Internet Technology Prof. Indranil Sengupta Department of Computer Science and Engineering Indian Institute of Technology, Kharagpur Lecture No #38 Search Engines and Web Crawler :: Part 1 Today we shall

More information

Link Analysis. CSE 454 Advanced Internet Systems University of Washington. 1/26/12 16:36 1 Copyright D.S.Weld

Link Analysis. CSE 454 Advanced Internet Systems University of Washington. 1/26/12 16:36 1 Copyright D.S.Weld Link Analysis CSE 454 Advanced Internet Systems University of Washington 1/26/12 16:36 1 Ranking Search Results TF / IDF or BM25 Tag Information Title, headers Font Size / Capitalization Anchor Text on

More information

Large-Scale Networks. PageRank. Dr Vincent Gramoli Lecturer School of Information Technologies

Large-Scale Networks. PageRank. Dr Vincent Gramoli Lecturer School of Information Technologies Large-Scale Networks PageRank Dr Vincent Gramoli Lecturer School of Information Technologies Introduction Last week we talked about: - Hubs whose scores depend on the authority of the nodes they point

More information

Web consists of web pages and hyperlinks between pages. A page receiving many links from other pages may be a hint of the authority of the page

Web consists of web pages and hyperlinks between pages. A page receiving many links from other pages may be a hint of the authority of the page Link Analysis Links Web consists of web pages and hyperlinks between pages A page receiving many links from other pages may be a hint of the authority of the page Links are also popular in some other information

More information

Introduction p. 1 What is the World Wide Web? p. 1 A Brief History of the Web and the Internet p. 2 Web Data Mining p. 4 What is Data Mining? p.

Introduction p. 1 What is the World Wide Web? p. 1 A Brief History of the Web and the Internet p. 2 Web Data Mining p. 4 What is Data Mining? p. Introduction p. 1 What is the World Wide Web? p. 1 A Brief History of the Web and the Internet p. 2 Web Data Mining p. 4 What is Data Mining? p. 6 What is Web Mining? p. 6 Summary of Chapters p. 8 How

More information

Desktop Crawls. Document Feeds. Document Feeds. Information Retrieval

Desktop Crawls. Document Feeds. Document Feeds. Information Retrieval Information Retrieval INFO 4300 / CS 4300! Web crawlers Retrieving web pages Crawling the web» Desktop crawlers» Document feeds File conversion Storing the documents Removing noise Desktop Crawls! Used

More information

The PageRank Computation in Google, Randomized Algorithms and Consensus of Multi-Agent Systems

The PageRank Computation in Google, Randomized Algorithms and Consensus of Multi-Agent Systems The PageRank Computation in Google, Randomized Algorithms and Consensus of Multi-Agent Systems Roberto Tempo IEIIT-CNR Politecnico di Torino tempo@polito.it This talk The objective of this talk is to discuss

More information

Information Retrieval. CS630 Representing and Accessing Digital Information. What is a Retrieval Model? Basic IR Processes

Information Retrieval. CS630 Representing and Accessing Digital Information. What is a Retrieval Model? Basic IR Processes CS630 Representing and Accessing Digital Information Information Retrieval: Retrieval Models Information Retrieval Basics Data Structures and Access Indexing and Preprocessing Retrieval Models Thorsten

More information

NBA 600: Day 15 Online Search 116 March Daniel Huttenlocher

NBA 600: Day 15 Online Search 116 March Daniel Huttenlocher NBA 600: Day 15 Online Search 116 March 2004 Daniel Huttenlocher Today s Class Finish up network effects topic from last week Searching, browsing, navigating Reading Beyond Google No longer available on

More information

Slides based on those in:

Slides based on those in: Spyros Kontogiannis & Christos Zaroliagis Slides based on those in: http://www.mmds.org A 3.3 B 38.4 C 34.3 D 3.9 E 8.1 F 3.9 1.6 1.6 1.6 1.6 1.6 2 y 0.8 ½+0.2 ⅓ M 1/2 1/2 0 0.8 1/2 0 0 + 0.2 0 1/2 1 [1/N]

More information

A New Technique for Ranking Web Pages and Adwords

A New Technique for Ranking Web Pages and Adwords A New Technique for Ranking Web Pages and Adwords K. P. Shyam Sharath Jagannathan Maheswari Rajavel, Ph.D ABSTRACT Web mining is an active research area which mainly deals with the application on data

More information

The PageRank Citation Ranking: Bringing Order to the Web

The PageRank Citation Ranking: Bringing Order to the Web The PageRank Citation Ranking: Bringing Order to the Web Marlon Dias msdias@dcc.ufmg.br Information Retrieval DCC/UFMG - 2017 Introduction Paper: The PageRank Citation Ranking: Bringing Order to the Web,

More information

SEO. Definitions/Acronyms. Definitions/Acronyms

SEO. Definitions/Acronyms. Definitions/Acronyms Definitions/Acronyms SEO Search Engine Optimization ITS Web Services September 6, 2007 SEO: Search Engine Optimization SEF: Search Engine Friendly SERP: Search Engine Results Page PR (Page Rank): Google

More information

CSE 494: Information Retrieval, Mining and Integration on the Internet

CSE 494: Information Retrieval, Mining and Integration on the Internet CSE 494: Information Retrieval, Mining and Integration on the Internet Midterm. 18 th Oct 2011 (Instructor: Subbarao Kambhampati) In-class Duration: Duration of the class 1hr 15min (75min) Total points:

More information

Administrative. Web crawlers. Web Crawlers and Link Analysis!

Administrative. Web crawlers. Web Crawlers and Link Analysis! Web Crawlers and Link Analysis! David Kauchak cs458 Fall 2011 adapted from: http://www.stanford.edu/class/cs276/handouts/lecture15-linkanalysis.ppt http://webcourse.cs.technion.ac.il/236522/spring2007/ho/wcfiles/tutorial05.ppt

More information

Weighted Page Rank Algorithm Based on Number of Visits of Links of Web Page

Weighted Page Rank Algorithm Based on Number of Visits of Links of Web Page International Journal of Soft Computing and Engineering (IJSCE) ISSN: 31-307, Volume-, Issue-3, July 01 Weighted Page Rank Algorithm Based on Number of Visits of Links of Web Page Neelam Tyagi, Simple

More information

Information Retrieval and Web Search Engines

Information Retrieval and Web Search Engines Information Retrieval and Web Search Engines Lecture 7: Document Clustering May 25, 2011 Wolf-Tilo Balke and Joachim Selke Institut für Informationssysteme Technische Universität Braunschweig Homework

More information

.. Spring 2009 CSC 466: Knowledge Discovery from Data Alexander Dekhtyar..

.. Spring 2009 CSC 466: Knowledge Discovery from Data Alexander Dekhtyar.. .. Spring 2009 CSC 466: Knowledge Discovery from Data Alexander Dekhtyar.. Link Analysis in Graphs: PageRank Link Analysis Graphs Recall definitions from Discrete math and graph theory. Graph. A graph

More information

Chapter 27 Introduction to Information Retrieval and Web Search

Chapter 27 Introduction to Information Retrieval and Web Search Chapter 27 Introduction to Information Retrieval and Web Search Copyright 2011 Pearson Education, Inc. Publishing as Pearson Addison-Wesley Chapter 27 Outline Information Retrieval (IR) Concepts Retrieval

More information

WEB SEARCH, FILTERING, AND TEXT MINING: TECHNOLOGY FOR A NEW ERA OF INFORMATION ACCESS

WEB SEARCH, FILTERING, AND TEXT MINING: TECHNOLOGY FOR A NEW ERA OF INFORMATION ACCESS 1 WEB SEARCH, FILTERING, AND TEXT MINING: TECHNOLOGY FOR A NEW ERA OF INFORMATION ACCESS BRUCE CROFT NSF Center for Intelligent Information Retrieval, Computer Science Department, University of Massachusetts,

More information

COMP5331: Knowledge Discovery and Data Mining

COMP5331: Knowledge Discovery and Data Mining COMP5331: Knowledge Discovery and Data Mining Acknowledgement: Slides modified based on the slides provided by Lawrence Page, Sergey Brin, Rajeev Motwani and Terry Winograd, Jon M. Kleinberg 1 1 PageRank

More information

Recommender Systems. Collaborative Filtering & Content-Based Recommending

Recommender Systems. Collaborative Filtering & Content-Based Recommending Recommender Systems Collaborative Filtering & Content-Based Recommending 1 Recommender Systems Systems for recommending items (e.g. books, movies, CD s, web pages, newsgroup messages) to users based on

More information

Developing Focused Crawlers for Genre Specific Search Engines

Developing Focused Crawlers for Genre Specific Search Engines Developing Focused Crawlers for Genre Specific Search Engines Nikhil Priyatam Thesis Advisor: Prof. Vasudeva Varma IIIT Hyderabad July 7, 2014 Examples of Genre Specific Search Engines MedlinePlus Naukri.com

More information

3 announcements: Thanks for filling out the HW1 poll HW2 is due today 5pm (scans must be readable) HW3 will be posted today

3 announcements: Thanks for filling out the HW1 poll HW2 is due today 5pm (scans must be readable) HW3 will be posted today 3 announcements: Thanks for filling out the HW1 poll HW2 is due today 5pm (scans must be readable) HW3 will be posted today CS246: Mining Massive Datasets Jure Leskovec, Stanford University http://cs246.stanford.edu

More information

Page rank computation HPC course project a.y Compute efficient and scalable Pagerank

Page rank computation HPC course project a.y Compute efficient and scalable Pagerank Page rank computation HPC course project a.y. 2012-13 Compute efficient and scalable Pagerank 1 PageRank PageRank is a link analysis algorithm, named after Brin & Page [1], and used by the Google Internet

More information

Lecture 9: I: Web Retrieval II: Webology. Johan Bollen Old Dominion University Department of Computer Science

Lecture 9: I: Web Retrieval II: Webology. Johan Bollen Old Dominion University Department of Computer Science Lecture 9: I: Web Retrieval II: Webology Johan Bollen Old Dominion University Department of Computer Science jbollen@cs.odu.edu http://www.cs.odu.edu/ jbollen April 10, 2003 Page 1 WWW retrieval Two approaches

More information

CS345a: Data Mining Jure Leskovec and Anand Rajaraman Stanford University

CS345a: Data Mining Jure Leskovec and Anand Rajaraman Stanford University CS345a: Data Mining Jure Leskovec and Anand Rajaraman Stanford University Instead of generic popularity, can we measure popularity within a topic? E.g., computer science, health Bias the random walk When

More information

Lecture 8: Linkage algorithms and web search

Lecture 8: Linkage algorithms and web search Lecture 8: Linkage algorithms and web search Information Retrieval Computer Science Tripos Part II Simone Teufel Natural Language and Information Processing (NLIP) Group Simone.Teufel@cl.cam.ac.uk Lent

More information

Project Problems SNU , Fall 2014 Chung-Kil Hur due: 12/21(Sun), 24:00

Project Problems SNU , Fall 2014 Chung-Kil Hur due: 12/21(Sun), 24:00 Project Problems SNU 4910.210, Fall 2014 Chung-Kil Hur due: 12/21(Sun), 24:00 Problem 1 (10%) Dust Storm The number of Mars exploration robots are more than one hundred now in A.D. 2032, and the number

More information

Bruno Martins. 1 st Semester 2012/2013

Bruno Martins. 1 st Semester 2012/2013 Link Analysis Departamento de Engenharia Informática Instituto Superior Técnico 1 st Semester 2012/2013 Slides baseados nos slides oficiais do livro Mining the Web c Soumen Chakrabarti. Outline 1 2 3 4

More information

Knowledge Discovery and Data Mining 1 (VO) ( )

Knowledge Discovery and Data Mining 1 (VO) ( ) Knowledge Discovery and Data Mining 1 (VO) (707.003) Data Matrices and Vector Space Model Denis Helic KTI, TU Graz Nov 6, 2014 Denis Helic (KTI, TU Graz) KDDM1 Nov 6, 2014 1 / 55 Big picture: KDDM Probability

More information

Some Interesting Applications of Theory. PageRank Minhashing Locality-Sensitive Hashing

Some Interesting Applications of Theory. PageRank Minhashing Locality-Sensitive Hashing Some Interesting Applications of Theory PageRank Minhashing Locality-Sensitive Hashing 1 PageRank The thing that makes Google work. Intuition: solve the recursive equation: a page is important if important

More information

Analytical survey of Web Page Rank Algorithm

Analytical survey of Web Page Rank Algorithm Analytical survey of Web Page Rank Algorithm Mrs.M.Usha 1, Dr.N.Nagadeepa 2 Research Scholar, Bharathiyar University,Coimbatore 1 Associate Professor, Jairams Arts and Science College, Karur 2 ABSTRACT

More information

2013/2/12 EVOLVING GRAPH. Bahman Bahmani(Stanford) Ravi Kumar(Google) Mohammad Mahdian(Google) Eli Upfal(Brown) Yanzhao Yang

2013/2/12 EVOLVING GRAPH. Bahman Bahmani(Stanford) Ravi Kumar(Google) Mohammad Mahdian(Google) Eli Upfal(Brown) Yanzhao Yang 1 PAGERANK ON AN EVOLVING GRAPH Bahman Bahmani(Stanford) Ravi Kumar(Google) Mohammad Mahdian(Google) Eli Upfal(Brown) Present by Yanzhao Yang 1 Evolving Graph(Web Graph) 2 The directed links between web

More information

An Adaptive Approach in Web Search Algorithm

An Adaptive Approach in Web Search Algorithm International Journal of Information & Computation Technology. ISSN 0974-2239 Volume 4, Number 15 (2014), pp. 1575-1581 International Research Publications House http://www. irphouse.com An Adaptive Approach

More information

Web Search Ranking. (COSC 488) Nazli Goharian Evaluation of Web Search Engines: High Precision Search

Web Search Ranking. (COSC 488) Nazli Goharian Evaluation of Web Search Engines: High Precision Search Web Search Ranking (COSC 488) Nazli Goharian nazli@cs.georgetown.edu 1 Evaluation of Web Search Engines: High Precision Search Traditional IR systems are evaluated based on precision and recall. Web search

More information

CRAWLING THE WEB: DISCOVERY AND MAINTENANCE OF LARGE-SCALE WEB DATA

CRAWLING THE WEB: DISCOVERY AND MAINTENANCE OF LARGE-SCALE WEB DATA CRAWLING THE WEB: DISCOVERY AND MAINTENANCE OF LARGE-SCALE WEB DATA An Implementation Amit Chawla 11/M.Tech/01, CSE Department Sat Priya Group of Institutions, Rohtak (Haryana), INDIA anshmahi@gmail.com

More information

Path Analysis References: Ch.10, Data Mining Techniques By M.Berry, andg.linoff Dr Ahmed Rafea

Path Analysis References: Ch.10, Data Mining Techniques By M.Berry, andg.linoff  Dr Ahmed Rafea Path Analysis References: Ch.10, Data Mining Techniques By M.Berry, andg.linoff http://www9.org/w9cdrom/68/68.html Dr Ahmed Rafea Outline Introduction Link Analysis Path Analysis Using Markov Chains Applications

More information

Website Name. Project Code: # SEO Recommendations Report. Version: 1.0

Website Name. Project Code: # SEO Recommendations Report. Version: 1.0 Website Name Project Code: #10001 Version: 1.0 DocID: SEO/site/rec Issue Date: DD-MM-YYYY Prepared By: - Owned By: Rave Infosys Reviewed By: - Approved By: - 3111 N University Dr. #604 Coral Springs FL

More information

Using Google s PageRank Algorithm to Identify Important Attributes of Genes

Using Google s PageRank Algorithm to Identify Important Attributes of Genes Using Google s PageRank Algorithm to Identify Important Attributes of Genes Golam Morshed Osmani Ph.D. Student in Software Engineering Dept. of Computer Science North Dakota State Univesity Fargo, ND 58105

More information

Chapter IR:II. II. Architecture of a Search Engine. Indexing Process Search Process

Chapter IR:II. II. Architecture of a Search Engine. Indexing Process Search Process Chapter IR:II II. Architecture of a Search Engine Indexing Process Search Process IR:II-87 Introduction HAGEN/POTTHAST/STEIN 2017 Remarks: Software architecture refers to the high level structures of a

More information

Mathematical Methods and Computational Algorithms for Complex Networks. Benard Abola

Mathematical Methods and Computational Algorithms for Complex Networks. Benard Abola Mathematical Methods and Computational Algorithms for Complex Networks Benard Abola Division of Applied Mathematics, Mälardalen University Department of Mathematics, Makerere University Second Network

More information

A brief history of Google

A brief history of Google the math behind Sat 25 March 2006 A brief history of Google 1995-7 The Stanford days (aka Backrub(!?)) 1998 Yahoo! wouldn't buy (but they might invest...) 1999 Finally out of beta! Sergey Brin Larry Page

More information

A P2P-based Incremental Web Ranking Algorithm

A P2P-based Incremental Web Ranking Algorithm A P2P-based Incremental Web Ranking Algorithm Sumalee Sangamuang Pruet Boonma Juggapong Natwichai Computer Engineering Department Faculty of Engineering, Chiang Mai University, Thailand sangamuang.s@gmail.com,

More information

Query Refinement and Search Result Presentation

Query Refinement and Search Result Presentation Query Refinement and Search Result Presentation (Short) Queries & Information Needs A query can be a poor representation of the information need Short queries are often used in search engines due to the

More information

Competitive Intelligence and Web Mining:

Competitive Intelligence and Web Mining: Competitive Intelligence and Web Mining: Domain Specific Web Spiders American University in Cairo (AUC) CSCE 590: Seminar1 Report Dr. Ahmed Rafea 2 P age Khalid Magdy Salama 3 P age Table of Contents Introduction

More information

Chapter 6: Information Retrieval and Web Search. An introduction

Chapter 6: Information Retrieval and Web Search. An introduction Chapter 6: Information Retrieval and Web Search An introduction Introduction n Text mining refers to data mining using text documents as data. n Most text mining tasks use Information Retrieval (IR) methods

More information

How to Get Your Website Listed on Major Search Engines

How to Get Your Website Listed on Major Search Engines Contents Introduction 1 Submitting via Global Forms 1 Preparing to Submit 2 Submitting to the Top 3 Search Engines 3 Paid Listings 4 Understanding META Tags 5 Adding META Tags to Your Web Site 5 Introduction

More information

PAGE RANK ON MAP- REDUCE PARADIGM

PAGE RANK ON MAP- REDUCE PARADIGM PAGE RANK ON MAP- REDUCE PARADIGM Group 24 Nagaraju Y Thulasi Ram Naidu P Dhanush Chalasani Agenda Page Rank - introduction An example Page Rank in Map-reduce framework Dataset Description Work flow Modules.

More information

An Overview of Search Engine. Hai-Yang Xu Dev Lead of Search Technology Center Microsoft Research Asia

An Overview of Search Engine. Hai-Yang Xu Dev Lead of Search Technology Center Microsoft Research Asia An Overview of Search Engine Hai-Yang Xu Dev Lead of Search Technology Center Microsoft Research Asia haixu@microsoft.com July 24, 2007 1 Outline History of Search Engine Difference Between Software and

More information

Domain Specific Search Engine for Students

Domain Specific Search Engine for Students Domain Specific Search Engine for Students Domain Specific Search Engine for Students Wai Yuen Tang The Department of Computer Science City University of Hong Kong, Hong Kong wytang@cs.cityu.edu.hk Lam

More information

Introduction to Search Engine Technology CS , Technion, Winter 2011/12

Introduction to Search Engine Technology CS , Technion, Winter 2011/12 Introduction to Search Engine Technology CS 236621, Technion, Winter 2011/12 Ronny Lempel Yahoo! Labs, Haifa What this Course is All About Will try to sample the science that drives Web search engines,

More information

The Security Role for Content Analysis

The Security Role for Content Analysis The Security Role for Content Analysis Jim Nisbet Founder, Tablus, Inc. November 17, 2004 About Us Tablus is a 3 year old company that delivers solutions to provide visibility to sensitive information

More information

Deep Character-Level Click-Through Rate Prediction for Sponsored Search

Deep Character-Level Click-Through Rate Prediction for Sponsored Search Deep Character-Level Click-Through Rate Prediction for Sponsored Search Bora Edizel - Phd Student UPF Amin Mantrach - Criteo Research Xiao Bai - Oath This work was done at Yahoo and will be presented as

More information

Chapter 2. Architecture of a Search Engine

Chapter 2. Architecture of a Search Engine Chapter 2 Architecture of a Search Engine Search Engine Architecture A software architecture consists of software components, the interfaces provided by those components and the relationships between them

More information

Data warehouses Decision support The multidimensional model OLAP queries

Data warehouses Decision support The multidimensional model OLAP queries Data warehouses Decision support The multidimensional model OLAP queries Traditional DBMSs are used by organizations for maintaining data to record day to day operations On-line Transaction Processing

More information

Jordan Boyd-Graber University of Maryland. Thursday, March 3, 2011

Jordan Boyd-Graber University of Maryland. Thursday, March 3, 2011 Data-Intensive Information Processing Applications! Session #5 Graph Algorithms Jordan Boyd-Graber University of Maryland Thursday, March 3, 2011 This work is licensed under a Creative Commons Attribution-Noncommercial-Share

More information

CHAPTER-27 Mining the World Wide Web

CHAPTER-27 Mining the World Wide Web CHAPTER-27 Mining the World Wide Web 27.1 Introduction 27.2 Mining the Web s Link Structure to Identify authoritative Web Pages 27.3 Automatic Classification of Web Documents 27.4 Construction of a Multilayered

More information

Breadth-First Search Crawling Yields High-Quality Pages

Breadth-First Search Crawling Yields High-Quality Pages Breadth-First Search Crawling Yields High-Quality Pages Marc Najork Compaq Systems Research Center 13 Lytton Avenue Palo Alto, CA 9431, USA marc.najork@compaq.com Janet L. Wiener Compaq Systems Research

More information

The PageRank Citation Ranking

The PageRank Citation Ranking October 17, 2012 Main Idea - Page Rank web page is important if it points to by other important web pages. *Note the recursive definition IR - course web page, Brian home page, Emily home page, Steven

More information

Information Systems & Semantic Web University of Koblenz Landau, Germany

<is web> Information Systems & Semantic Web University of Koblenz Landau, Germany Information Systems & University of Koblenz Landau, Germany Semantic Search examples: Swoogle and Watson Steffen Staad credit: Tim Finin (swoogle), Mathieu d Aquin (watson) and their groups 2009-07-17

More information

KEY FUNCTIONS. Dynamic content. Conditional s. A/B/X testing. Creation. testing. Video s. Spam. testing. Advanced personalisation

KEY FUNCTIONS. Dynamic content. Conditional  s. A/B/X testing. Creation. testing. Video  s. Spam. testing. Advanced personalisation KEY FUNCTIONS Dynamic content Conditional emails Messages sent as part of one campaign will automatically adjust their looks to particular addressees preferences. Depending on the values of the characteristics

More information

SEO 1 8 O C T O B E R 1 7

SEO 1 8 O C T O B E R 1 7 SEO 1 8 O C T O B E R 1 7 Search Engine Optimisation (SEO) Search engines Search Engine Market Global Search Engine Market Share June 2017 90.00% 80.00% 79.29% 70.00% 60.00% 50.00% 40.00% 30.00% 20.00%

More information

Focused crawling: a new approach to topic-specific Web resource discovery. Authors

Focused crawling: a new approach to topic-specific Web resource discovery. Authors Focused crawling: a new approach to topic-specific Web resource discovery Authors Soumen Chakrabarti Martin van den Berg Byron Dom Presented By: Mohamed Ali Soliman m2ali@cs.uwaterloo.ca Outline Why Focused

More information