Web Search Algorithms - 1 -

Transcription

1 Web Search Algorithms - 1 -

2 Why web search in this module? WWW is the delivery platform and the interface How do we find information and services on the web we try to generate a url that seems sensible Dell Computers Ford Ireland But products? GPS Devices is not ok Or, we use a Search Engine So we rely on Search Engines - we even use them to look up spellings and as a calculator! Search Engines bring people to a website For most, such as Google, ranking algorithm is closely guarded, wholesome, true, uncorrupted, and not paid advertisements are merely sold based on similarity to query keywords. This leads to the industry of Search Engine Optimisations (SEO)... the Google Dance - 2 -

3 Text IR - Google as example Google is operational since 1998 Two PhD students from Stanford?? Billion documents Early search engines competed on size of index, related to how powerful their infrastructure was. Not an issue now. Stopped advertising after 8,168,684,336 pages in Aug 2005 Size now, effectively unknown Also has??? billion images not all unique images Flickr has about 2B (Nov 2007); FaceBook had 4.1 B at that time - 3 -

4 Searching or Marketing? However, Search Engines must make a profit! Advertisment Sales Marketing Paid Listings And selling their indexes A lot of Search Engines are also marketing companies This is at odds with the idea that a search engine is a page you visit on the way elsewhere. The less time you spend there the better! But, many people pass through the doors, so they sell query focussed advertisements You can estimate by looking at the main page of the search engine

5 How do SEs help user searches It is known that we search for people / home pages. companies / company HPs (or guess from URLs). a particular product or service. a fact, buried in one or more documents, any one of which will do a document, an entire document, with text/image, and nothing smaller will do. an overview on a broad or narrow topic Media Search an MPEG-4 file. Through image databases. Through (digital) video library, and/or through a video. If the SE knows the type of query, then ranking can be tailored to that query, because different search types can be satisfied by different search algorithms

6 Search Engines Originally SE s were web directorys Manually generated (e.g. Yahoo!) Then automatic crawler-based Search Engines developed The web got big and manual categorisation was becoming too difficult (e.g. Lycos) Today the large SE s index over?? billion web pages. The first crawler-based SE was the WWWW in

7 Architecture of a Search Engine - 7 -

8 My Google! - 8 -

9 Bing -9-

10 Facebook? Is it a Search Engine?

11 Facebook Social Graph A Previous Class College Friends Friends IR Research Community

12 TWITTER

13 The Landscape is changing

14 Web 1.0 Web 3.0 Web 1.0 Static content... Companies created content We were consumers Web 2.0 User generated content Communities and creators... We create, filter, recommend the content Web 3.0 UGC and... Semantic Web... Life streams? Social and Location What is the next big thing?

15 Web 1.0 Search engines over prepared and planned content Organisations and some users SEO was the way to optimise WEB 1.0 HTML and static content

16 - 16 -

17 Web 2.0 User and Organisation Generated Content Social Graphs Social Filtering and Social Ranking Examples: Social networks : facebook, twitter, linkedin Shared bookmarks: digg, delicious, reddit, stumbleupon Social media sharing :flickr, youtube Blogs (MSN space, wordpress, blogger) Even 3D social worlds... Social gaming?

18 - 18 -

19 Web 3.0 Semantic Web Many media types... Integrated for smarter uses Rich media integration Personalisation to the user context Life streaming of content We are integrated into our own entertainment

20 What is Web3.0 about?

21 The Search Landscape Changing enormously

22 Continuous Partial Attention Be aware of Continuous Partial Attention... a kind of multitasking skimming the surface of the incoming data, picking out the relevant details, and moving on to the next stream. Continuous not episodic Cast a wider net, but never full attention So.. How does this impact on search?

23 And don t forget the twitter curve

24 Google AdSense

25 Spamming Spamming is a technique based on the manipulation of content in order to affect ranking from search engines Bogus meta tags, hidden text, plan text Also link spamming Huge SE resources are used in defeating spamming - more than in search quality improvement! Getting in the top-10 is essential for businesses 85% of users only look at top 10. Lead to the business of Search Engine Optimisation

26 Search Engine Ranking As we all know, simply examining web page content as text is not enough.. We need to examine ranking factors.. Positive and negative

27 Positive Ranking Factors : Term Location In the TITLE of the page, most important In the body of the text, but must MAKE SENSE In the Heading text (H1,H2 ) In the Domain Name Also in page URL In ALT tag and image title In BOLD/STRONG tags Terms near the top likely ranked higher than other terms

28 Positive Ranking Factors : Page Attributes Importance of the page in the Website Number of links to it from the same website Quality of links to other pages Age of a document Older may be more authorative We will see authorities later! Newer may be better for some queries (e.g. news) Amount of text on the page Structure of the page Frequency of updates Spelling and correctness of HTML

29 SE Ranking + : Website Issues Linkage of the Website Global link popularity of the website Like a global Pagerank (SiteRank) Relevance of the links into the website Link popularity of the site in a topical community Rate of new inbound links to a website Age of a website (older is better) Freshness of a website (new pages is better) Relevancy of the website (as well as the page) Clickthrough rate for the website Reputation of the top-level domain E.g..GOV &.EDU can not easily be bought

30 SE Ranking + : Linkage Issues Anchor text of inbound links as a description of the WWW page Also text surrounding the link into the webpage Topical relationship between source and target of link Link popularity of the page in a topical community Age of links The older the better, i.e. long lasting links Pagerank of the webpage Googles PageRank algorithm Number of links into a web page

31 Positive Ranking Factors : Images Images on a web page Can provide a chance to express ideas in a visual way that can convey a considerable amount of information Add to the attractiveness and perceived quality of a site. Recent Microsoft Patent on Scoring Relevance of a Document Based on Image Text Also.. Remember to name the image properly and have alt element

32 Negative Ranking Factors Link Farm Participation Try to artificially increase PageRank Proportion of links to or from known Spamming sites Duplicate Content to already indexed content Server Errors or server down-time External links to low-quality content Low level of visitors to the website Try to include hidden text on the page

33 Using the Ranking Factors PageRank Factors Linkage Factors Negative Factors Website Factors Page Factors Term Location Factors Result User Query The Search Engine ranking process is a closely guarded trade secret of the search engines

34 So lets look in some detail at some of these ranking factors Linkage-based Search

35 The Shape of the WWW This is based on a study of 200 million web pages. Scale up to WWW scale

36 Spidering : finding WWW content A Search Engine needs to find WWW content for its index This is done by the spidering software Starting from some seed WWW pages, the spider software downloads these pages and extracts the links, thereby learning about new pages to crawl. WWW-scale crawling means crawling thousands of pages per second

37 A Basic Crawling Algorithm You need to be linked to from the main Remember the shape! Given a set of seed URLs (WWW pages addresses): Add them to a (priority) queue of URLs While the queue is not empty (!empty) Take the first URL (u) off the queue Download the WWW page for u Store the URL in a list of seen URLs Index it If u is a HTML page, extract the links (y) For each y add it to the queue if it has not been visited before

38 Spiders must behave! Most crawlers/spiders will follow some rules: A spider must never request large numbers of documents from the same host sequentially change the target website as often as is feasible. A spider must never (for whatever reason) repeatedly request the same document. If a document is unavailable, it s position in the queue must be penalized Repeated failures must be taken into account and the document flagged as unavailable and taken off the queue. A spider must respect author s wishes as expressed using the robots exclusion protocol

39 Robots Exclusion allows Web site administrators to indicate to visiting robots which parts of their site should not be visited by the robot. Most good robots will process it BUT it makes a crawler less efficient more explorative crawling required To exclude all robots from the entire server User-agent: * Disallow: / To allow all robots complete access User-agent: * Disallow: To exclude all robots from part of the server User-agent: * Disallow: /cgi-bin/ Disallow: /private/ To exclude a single robot User-agent: BadBot Disallow: / To allow a single robot User-agent: WebCrawler Disallow: User-agent: * Disallow: /

40 Robots.txt example

41 Another Example

42 And one more

43 Simple Overview 1. Spidering WWW 2. Indexing View WWW page 3. Ranking

44 WWWW the first SE WWWW (94) did not use the content of a page for indexing, it used: Title of the document Text in the URL String Any anchor text from links pointing to the page. Based on using the UNIX egrep program to search through disk files. All SEs now use Linkage Analysis to exploit latent human judgement to improve retrieval performance This is in addition to using the document content

45 Some history Citation Analysis Most significant contribution to web search is the technique for how to rank Journals based on quality (impact) Citation indexing the impact factor measurement based on two elements: the number of citations in the current year to any articles published in the journal over the previous two years. the number of articles published by the journal during these two years. Letting j be a journal and IF j be the Impact Factor of journal j, we have: # Citations ( last 2years ) IF j = # Published Articles ( last 2years ) This impact factor was originally applied to medical journals as a simple method of comparing journals to each other regardless of their size

46 Hirsch Index (h-index) Citation Analysis is a balance between quality (number of citations) and quantity (number of papers); Among scientists, the h-index is becoming popular for measurement it s the number of published papers which each have a number of citations greater or equal to that number. Alan Smeaton has 250+ papers, about 3,000 citations, and an h-index of 30; Desmond Higgins (UCD) has 29,000 citations (22,500 on one paper), and an h-index of 22; Linkage analysis in web topology does something like this, as we ll see

47 Linkage Analysis Linkage Analysis : a method of ranking web sites which is based on the exploitation of latent human judgments mined from the hyperlinks that exist between documents on the WWW. The first generation of web search engines were effectively TF-IDF or BM25, or equivalent. And they have addressed the engineering problems of web spidering and efficient searching for large numbers of both users and documents. Linkage Analysis important since late 90s. Anecdotally this appears to have improved the precision of retrieval yet there was little scientific evidence in support of this until recently

48 Origin : Citation Analysis How to rank Journals based on quality (impact) Citation indexing the impact factor measurement based on two elements: the number of citations in the current year to any articles published in the journal over the previous two years. the number of articles published by the journal during these two years. Letting j be a journal and IF j be the Impact Factor of journal j, we have: # Citations ( last 2years ) IF j = # Published Articles ( last 2years ) This impact factor was originally applied to medical journals as a simple method of comparing journals to each other regardless of their size

49 Mining links can tell us that Bibliographic Coupling A and B are similar because they both cite C,D,E Co-citation Analysis A and B are similar because they are both cited by C,D,E

50 What else can we do with links? Count them? Distinguish between good and bad ones? How we employ them is called Linkage Analysis Linkage-based ranking schemes can be seen to belong to one of two distinct classes: Query-independent schemes, A score is assigned to a document once and used for all subsequent queries.» independent of a given query. Fast processing at query time! Query-dependent schemes, assigns a linkage score to a page in the context of a given query. Slower processing at query time!

51 Assumed Properties of Links When extracting information for linkage analysis from hyperlinks on the Web, two core properties can be assumed: A link between two documents on the web carries the implication of related content. If different people authored the documents (different domains, therefore off-site links), then the first author found the second document valuable. An author can-not be allowed to influence the linkage score of documents within his/her domain. Off-site links (links between web sites) are more important that links within websites or within documents

52 Link Types in-link to doc F : 5,8,9 out-link from doc F : 4,6,10 self-links: 2,11 on-site links: 6,8,12 off-site links: 1,3,4,5,9,10 on-site in-links to doc F:? off-site out-links of doc F:?

53 Basic Linkage Analysis Given a linkage graph (below), Page A is a better page than B because F G Off-site links only C A H B J D E I

54 Expanding on this However, page B may actually be better F G C A H B J CNN E Yahoo I D So we use iterative processes like PageRank or Kleinberg s

55 Generating a linkage score Let n be some web page and S n be the set of web pages that link into n across off-site links: P n = S n In this case, the P n score (Popularity score) is based purely on the in-degree of document n Could be the sole source of document ranking given a set of relevant documents (boolean IR) OR could work by integrating normal document retrieval (TF-IDF / BM25 scores) to generate an overall weight. Once again, we let n be some web page and S n be the set to pages that link into n: (#! Sim( q, n) ) + ( ) Sc ' = "! n S n parameters assumes normalisation

56 More simple linkage techniques Weighted Citation Raking Spreading Activation & Co-citation Analysis SA: Spreads a score across outlinks CA: Passes a score back to hub document

57 Hubs & Authorities A Hub is a document that contains links to many other documents C D E F An Authority is a document that many documents link to W A good Hub links to good Authorities A X A good Authority links to good Hubs Y Z

58 What makes a good Hub? What makes a good hub for the query web browsers? Internet Explorer Amaya Netscape Opera Hub Mozilla Firefox NeoPlanet MyBrowser

59 What Makes a good Authority What makes a good Authority for the query web browsers? Hub Hub Hub Hub Hub Hub Hub Internet Explorer Amaya Hub Hub Netscape Mozilla Hub Opera Firefox NeoPlanet MyBrowser

60 And What makes these authorities good? Good hubs that themselves link into good authorities a self-re-inforcing relationship! Hub Hub Hub Hub Hub Hub Hub Internet Explorer Amaya Hub Hub Netscape Mozilla Hub Opera Firefox NeoPlanet MyBrowser

61 The Influence of Links A Document s content can be represented by the anchor text of the in-links (all) into that doc, not by the document itself. More in-links, means more content, better chance of getting returned for a query. Very Simple, but effective! Improved by windowing Document Anchor Text Doc

62 The Importance of Windows

63 The Importance of Windows

64 Iterative Linkage Algorithms PageRank

65 PageRank Query INDEPENDENT score for every documents An important aspect of Google ranking? It allocates a PageRank (query independent importance) score to every document in an index, and this score is used when ranking documents. Simple Iterative Algorithm Until convergence A simulation of a random user s behaviour when browsing the web. Equivalent to a user randomly following links, or getting bored and randomly jumping to a random page anywhere on the WWW. In effect it is based on the probability of a user landing on any given page. This can be applied to other graphs than the WWW graph social networks, blog comments?

66 Key points The PR of A is divided equally among its outlinks 1/4 The PR of B is equal to the sum of the transferable PR of all its in-links W PR W =1 1/4 PR A = 1 A X PR X =1 B PR B = 2¼ 1/4 1/4 Y PR Y =1 Z PR Z =1 ¼ ½ ½ + 1 2¼

67 For Example the PageRank PR F of document F is equal to PR B divided the outdegree of B summed with PR D divided by the out-degree of D. PRB PR F = + 2 PR 3 D

68 The Simplified Technique 1, Calculate a pre-iteration PageRank score for each document for all n in N, PR = 2, Calculate PageRank score for each document n 1 N PR' n = c #! m" S n outd PR m egree m assume c = 1 3, Store new PageRank scores for all n in N, PR = PR n ' n 4, If not convergence then goto

69 A Simple Web Graph F G A B C E D

70 PageRank Sample Graph Total =

71 PageRank after Iteration Total =

72 PageRank after Iteration Total =

73 PageRank Problem 1 (Dangling Links)??

74 PageRank - Problem 2 (Rank-Sink)

75 PageRank Problem 1 (Dangling Links)??

76 PageRank Problem 1 (Dangling Links) removed

77 PageRank - Problem 2 (Rank-Sink)

78 PageRank - Problem 2 (Rank-Sink) 15% 15% 15% A Vector over All Web Pages 0.14 Doc Doc 2 15% 15% Doc 3 Doc 4 Doc Doc Doc 7 15% 15% Hence if all PageRanks sum to 1.0, then E =

79 The two problems Dangling Links: these are links that point to a page which itself contains no outlinks Docs which the system knows about (and has anchor text descriptions for) but has not downloaded yet. Or just docs with no links out If the PageRank of the web pages associated with the target of these links is not redistributed at each iteration and is lost from the system SOLUTION : Remove page or use Universal Document Rank Sinks: these are two or more pages that have outlinks to each other, but to no other pages. Assuming we have at least one inlink into these pages from a page outside of these pages then at each iteration rank enters these pages and never exits accumulates rank SOLUTION: using the E Vector with E = 0.15 or the inclusion of a Virtual (Universal Document)

80 How to use this Vector? This vector has an entry for each document and is used as an indicator of how to distribute any redundant rank back into the system. Each documents entry in the Vector (E) represents the proportion of rank to given to that document, but it is believed to be uniform with E = 0.15 if the sum of all pageranks sums to 1. But we can do personalisation e.g. to focus on Formula1 pages increase their weight in E. Letting E n be some vector over the Web pages that corresponds to a source of rank, c is a constant which is maximised and PR = 1 (sum of all PageRanks = 1), we have the following formula: PR m PR ' n c! + (1 " c)! m$ Sn outd egreem = # E n

81 Alternate Solution! Vector 0.14 Doc Doc 2 UD * Doc 3 Doc Doc Doc Doc 7 Probability of a user being bored is now 1/(n+1) where n = number of outlinks not

82 Personalised PageRank Vector 0.10 Doc Doc 2 UD * Doc 3 Doc Doc Doc Doc

83 Using PageRank Query PageRank Array Content Score (n) PageRank Score (n)??? Formula??? Final Document Score

84 Kleinberg s Algorithm Kleinberg s algorithm is similar to PageRank, in that it is an iterative algorithm based purely on the linkage of the documents on the web. However it does have some major differences: It is executed at query time, and not at indexing time, with the associated hit on performance that accompanies query-time processing. Is it used in SE s not common! It computes two scores per document (hub and authority) as opposed to a single score. It is processed on a small subset of relevant documents, not all documents as was the case with PageRank

85 Recall Hubs and Authorities HUB Page: a hub page is a page that contains a number of links to pages containing information about some topic, e.g. a resource page containing links to documents on a topic such as Formula 1 motor racing. Required pages have a hub score representing it s quality as a source of links. AUTHORITY Page: an authority page is one that contains a lot of information about some topic, an authoritive page. Consequently, many pages will link to this page, thus giving us a means of identifying it. Required pages also have an authority score representing its perceived quality by other people. Documents with high authority scores are expected to contain relevant content, whereas documents with high hub scores are expected to contain links to relevant documents

86 HITS Process 2 1 Root Set Expanded Set 3 ( Authq ) ( Hub ) Hubp =! for all p " q Authp =! q for all q " p Focused subgraph of WWW

87 Hub Scores Hub p ( Auth ) =! q for all p " q X P Y Z Q contains X,Y and Z

88 Authority Scores Auth p ( Hub ) =! q for all q " p X Y P Z Q contains X,Y and Z

89 Kleinberg s HITS Technique Iteratively calculates Hub & Authority scores Begin with all Hubs & Authority scores = iterations needed until convergence Hub scores based on Authority scores of off-site outlink docs. Auth scores based on Hub scores of off-site inlink docs. Return top X Hubs and/or Authorities Once expanded set generated then no further content analysis (Topic Independent). Narrow Topic will diffuse to a Broader Topic Broad Topic may produce inaccurate results

90 Kleinberg s Algorithm Hubi # 1 Authi # 1 loop : for n = 1,2,... N : Auth' Hub' Normalise Auth' Normalise Hub' end while( not converged ) n n = =! m" S! n" T n n Hub Auth n n, m o, obtaining Auth obtaining Hub n n

91 Wrapping up SEs SE s now provide more than just searching and are portals - consumer-oriented gateway to web resources which is editorially controlled links to what search engines, or their paying clients, believe you may be interested in. Search engines are for profit ventures, not charities Some sell their indexes Mostly advertising 10% to 15% of queries to the major search engines are on adult themes. Offer lots of extra s including : media search, identification of names, Amazon links, related searches listing, page translation, language specific search then there is photo management, , music

92 Final thoughts Sub 1 second querying is essential No time for interesting algorithms, Q&A, Manual Query Expansion, Belief is that searchers happy with sub-optimal results as long as no delay in getting them. No industry standard benchmark for evaluation