HOW SEARCH ENGINES WORK THE WEB IS A DIRECTED GRAPH CS 115: COMPUTING FOR THE SOCIO-TECHNO WEB FINDING INFORMATION WITH SEARCH ENGINES. User.

Transcription

1 CS 115: COMPUTING FOR THE SOCIO-TECHNO WEB FINDING INFORMATION WITH SEARCH ENGINES THE WEB IS A DIRECTED GRAPH The first-ever World Wide Web site went online in 1991, although this doesn t seem that long ago, it is hard to imagine the world before Sir Tim Berners-Lee s invention. In many ways, the colossal impact of the World Wide Web is obvious. Many people, however, may not fully appreciate the underlying technical contributions that make the Web possible. Sir Tim Berners-Lee not only developed the key components, such as URIs and web browsers that allow us to use the Web, but offered a coherent vision of how each of these elements would work together as part of an integrated whole. ACM President Vicki L. Hanson HOW SEARCH ENGINES WORK Like a map of a country with cities and one-way roads Sponsored Links User A n Nodes = web pages Arcs = hyperlinks from a page to another Why is this cool? Because it can be explored it can be indexed Miele Vacuum Cleaners Miele Vacuums- Complete Selection Free Shipping! Miele Vacuum Cleaners Miele-Free Air shipping! All models. Helpful advice. Directed Graph of Nodes and Arcs (one-way connections) n CG Appliance Express Discount Appliances (650) Same Day Certified Installation San Francisco-Oakland-San Jose, CA Web B Results 1-10 of about 7,310,000 for miele. (0.12 seconds) Miele, Inc -- Anything else is a compromise Web spider At the heart of your home, Appliances by Miele.... USA. to miele.com. Residential Appliances. Vacuum Cleaners. Dishwashers. Cooking Appliances. Steam Oven. Coffee System k - Cached - Similar pages Miele Welcome to Miele, the home of the very best appliances and kitchens in the world k - Cached - Similar pages Miele - Deutscher Hersteller von Einbaugeräten, Hausgeräten... - [ Translate this page ] Das Portal zum Thema Essen & Geniessen online unter Miele weltweit...ein Leben lang.... Wählen Sie die Miele Vertretung Ihres Landes k - Cached - Similar pages Herzlich willkommen bei Miele Österreich - [ Translate this page ] Herzlich willkommen bei Miele Österreich Wenn Sie nicht automatisch weitergeleitet werden, klicken Sie bitte hier! HAUSHALTSGERÄTE k - Cached - Similar pages E C Search Indexer D The Web Indexes Ad indexes

2 PAGERANK: GOOGLE S PRIDE The reputation PageRank PR(W) of a page W = the sum of a fair fraction of the reputations PR(W j ) of all pages W j that point to W Beautiful Math behind it PR(W ) = PR(W 1) O(W PR equivalent to the chance 1 ) + PR(W 2) O(W 2 ) +...+PR(W n) O(W n ) of randomly surfing to the page W1 PR idea similar to academic co-citations How to compute PR: Each page starts with some basic reputation (e.g., = 1) and repeatedly distributes fair (equal) fractions of reputation to its linked pages (while receiving fair fractions from others) until equilibrium (no further changes occur) W2 W3 W. PAGERANK: ITERATIVE COMPUTATION PR(W ) = PR(W 1) O(W 1 ) + PR(W 2) O(W 2 ) +...+PR(W n) O(W n ) Set initial PR values to 1 Solve the following equations iteratively: PR(A) = PR(C) PR(B) = PR(A) / 2 PR(C) = PR(A) / 2 + PR(B) IT IS SLIGHTLY MORE COMPLICATED PAGERANK: ITERATIVE COMPUTATION The reputation PageRank PR(W) of a page W = t/n + (1-t) * the sum of a fair fraction of the reputations PR(W j ) of all pages W j that point to W PR(W ) = t N + (1 t)( PR(W 1 ) O(W 1 ) + PR(W 2 ) O(W 2 ) +...+PR(W n ) O(W n ) ) W is a web page Wi are the web pages that have a link to W O(Wi) is the number of out-links from Wi t is the teleportation probability (the chance that we may visit a page randomly) N is the size of the Web (that we have seen) W2 W1 W3 W. PR(W ) = t N + (1 t)( PR(W 1) O(W 1 ) + PR(W 2) O(W 2 ) +...+PR(W n ) O(W n ) ) t is normally set to 0.15, but for this example, for simplicity let s set it to 0.5 Set initial PR values to 1 Solve the following equations iteratively: PR(A) = 0.5/3+ 0.5PR(C) PR(B) = 0.5/3+ 0.5(PR(A) /2) PR(C) = 0.5/3+ 0.5(PR(A) /2 + PR(B))

3 WHAT ARE YOU TRYING TO FIND? Types of queries: Informational want to learn about something Peripheral neuropathy Navigational want to go to that page Wellesley College Transactional want to do something (web-mediated) Access a service Downloads Shop Gray areas Find a good hub (resource collection) Exploratory search see what s there Wellesley weather Mars surface images Nikon SLR camera car rental Boston morality of abortion HOW FAR DO YOU LOOK FOR RESULTS? TRENDING TOPICS QUESTIONS ABOUT THE WEB How big is the Web? How many people use the Web? How many people use search engines? How hard is it to go from one page to another through clicks? What is the shape of the Web?

4 HOW BIG IS THE WEB? Number of accessible web pages (the visible web) Google claims to have encountered 1 trillion unique URLs (though in the past claimed to have indexed 26.6 billion pages Yahoo claims to have indexed 55 billion pages Cuil claims to have indexed 120 billion pages The deep web (or hidden or invisible web) contains times more information Coverage (i.e. the proportion of the web indexed) is crucial for search engines. Today, less than 15% pages are indexed! HOW MANY PEOPLE USE SEARCH ENGINES? 49% of all internet users use a search engine on a daily basis 6,586,013,574 searches a day worldwide (August 2016) Search engine usage as of June 2004: Google (41.6%), Yahoo! (31.5%), MSN (27.4%), AOL (13.6%), Ask (7%) Search engine usage as of March 2017:? What does this tell you about the importance of the Search Engines? HOW HARD IS IT TO SURF FROM ONE PAGE TO ANOTHER? WHAT IS THE SHAPE OF THE WEB? Over 75% of the time there is no directed path from one random web page to another. When a directed path exists its average length is 16 clicks. Short average path between pairs of nodes is characteristic of a small-world network. Map of the Internet (1998)

5 WHAT IS THE SHAPE OF THE WEB? STRONGLY CONNECTED COMPONENT Bow-tie shape of the web strongly connected component (SCC) in a directed graph is a subset of the nodes such that: (i) every node in the subset has a path to every other; and (ii) the subset is not part of some larger set with the property that every node can reach every other. BOWTIE TERMINOLOGY: LARGEST SCC, CORE, IN, OUT, ISLANDS, TENDRILS EXERCISES 1-Draw a web graph of the course class website. 2-Draw a web graph of the MAS program website.

6 OPTIONAL MATERIAL A CONSTRUCTIVE ALGORITHM TO PROVE THAT THE WEB IS A BOWTIE Based on the paper Why is the shape of the Web a Bowtie? Start with disconnected Web pages Examine the shape after 1 link/page is considered Bowtie appears after the 2 nd link per page is considered After that, the Bowtie shape gets stronger AFTER ONE LINK IS CONSIDERED AFTER A SECOND LINK IS CONSIDERED A collection of pseudo-trees A collection of bowties

7 WHEN MORE LINKS ARE INCLUDED Consider the combinations of links within the same bowtie between bowties CORRECT THE SHAPE OF THE WEB Bowties are everywhere! HOW ABOUT THE CLASS WEB? CAN WE COVER ALL THE WEB? Crawling starting point Crawling starting point Put a starting Web page in a queue Q & repeat: Pick up a page P from the queue, Crawl P, and Put on the queue each page reachable from P

8 WEB DIRECTORIES ORGANIZE INFORMATION IN CATEGORIES WITH HUMAN HELP WHAT PEOPLE ARE SEARCHING FOR? MECHANICS OF A TYPICAL SEARCH