Web Information Retrieval. Chapter 1. Motivation & Overview

Transcription

1 Institute for Web Science & Technologies - WeST Why are you here? Web Information Retrieval Dr. Dr. Summer Term 2013 S UMMERA CADEMY 2 What is Web Retrieval all about?.. discovering useful informa2on from the World- Wide Web and its usage pa9erns using Text Mining: w applica2on of Data Mining techniques to unstructured text (e.g. Web sites, blog pos2ngs, user comments..) Structure Mining: w taking into account structure and rela2ons in Web data (HTML tags, hyperlinks, friend lists..) Usage Mining: w taking into account user interac2ons with Web systems (clickstreams, collabora2ve filtering,...) Chapter 1 Motivation & Overview 3 4 Theoretic model of IR How do users seek information Information Need Formulate query Send query Retrieve results Evaluate results satisfied? Simple, linear model w Information need remains constant w Feedback to clarify information need Underlies most current systems Not realistic! Concept: Anomalous State of Knowledge (ASK) Done! Fuhr [8] Belkin [2] 5 6 1

2 Example: Web exploration? Query: A famous car manufacturer born near Koblenz 7 8 Example: Web exploration Query: A famous car manufacturer born near Koblenz Answer: August Horch, born 1868 in Winningen 9 10 Berrypicking model Bates [1]

3 Homework: Web exploration task Types of queries Type? Query: A famous furniture manufacturer born near Koblenz Who promoted him? What kind of product made him so famous? Intention Example 1 Navigational Go to specific known website wikipedia 2 Informational Learn something by reading web pages 2.1 Directed Learn something in particular about my topic Closed Get an single, unambiguous to a question summer academy lectures Open Get an answer to an open-ended question effective Information retrieval models 2.2 Undirected Learn anything/everything about a topic Information retrieval 2.3 Advice Get advice, ideas, suggestions, instructions. Implement sorting algorithm 2.4 Locate Find real world service or product phone card 2.5 List List of web sites helping me achieve some goal travel 3 Resource Obtain a resource (not information) available on web 3.1 Download Download a resource for use on computer ebooks 3.2 Entertainment Be entertained by viewing items live camera in koblenz 3.3 Interact Interact with a program/service available on the web measure converter 3.4 Obtain Obtain a resource that does not require a computer to use. summer academy time table Rose, Levinson [20] 13 How users use the Web? 14 Where are the users? 70-80% of users use SE to find sites! and most users prefer a few commercial large-scale search engines Google: the users are all over the world Search engine serves over 100 different languages Should not have a catastrophic failure in any What are the users asking us for? Google-style Web search: Supporting the user: Result representations Users give a 2-4 word query SE gives a relevance ranked list of web pages Most users click only on the first few results Few users go below the fold.. whatever is visible without scrolling down Far fewer ask for the next 10 results over 200 Million queries a day noisy inputs searching over Eight Billion+ documents

4 Organization of Search Results Organization of Search Results (2) cluster search results into topic areas large-scale Web search with authority ranking 19 Organization of Search Results (4) 20 Organization of Search Results (5) show broader context of results Tag cloud of related terms 21 Organization of Search Results (6) 22 Organization of Search Results (7) suggest related search directions Meta search with previews

5 Organization of Search Results (8) Organization of Search Results (9) Preview based browser of results Manual categorization of cites Under the hood: crawling and indexing How should it work? Web IR in a nutshell 27 Crawling Metternich: One of the famous Austrian politicians Extraction of relevant words web Austrian Politicians... Linguistic methods: stemming Thesaurus (Ontology) e.g. synonyms, sub-/superconcepts Metternich = district of Koblenz? Koblenz =city in Rhineland-Palatinate? RP = federal state in Germany? metternich austria politician... Statistically weighted features (terms) Indexing Index (e.g. B + -tree) politician austria... URLs 28 Observation: Search engines have different data Observation: they also show different results Source: A. Gulli, A. Signorini, WWW 2005 Google > 8 Bio., MSN > 5 Bio., Yahoo! > 4 Bio., Ask/Teoma > 2 Bio. overlap statistics (surface) Web > 11.5 Bio. pages (> 40 TBytes) Deep Web (Hidden Web) estimated to have 500 Bio. units (> 10 PBytes)

6 Ranking: Content Relevance Ranking by descending relevance Search engine sim (d,q): = F Query q [0,1] (Set of weighted features) F Documents are feature vectors di [0,1] Similarity metric: i F j1 = d q ij F F 2 2 dij qj j1 = j1 = j PageRank in a nutshell random walk on the Web graph: uniformly random choice of links + random jumps PR( q ) = ε j( q ) + ( 1 ε ) PR( p ) t( p,q ) p IN(q) e.g., using: 2 dij : = wij / k wik freq( f j, di ) # docs wij := log maxk freq( fk, di ) # docs with fi tf*idf formula Authority (page q) = stationary prob. of visiting q Ranking: link-based authority What is our technology used for? Ranking by descending relevance & authority Search engine crawl extract & clean index search rank present F Query q [0,1] (Set of weighted features) Additionally, consider links between Web nodes: Authority Score (d i ) := stationary visit probability [d i ] in the random walk on the Web.. reconciliation of relevance and authority by ad hoc weighting 33 strategies for crawl schedule and priority queue for crawl frontier handle dynamic pages, detect duplicates, detect spam build and analyze Web graph, index all tokens or word stems special file system for high- index-entry and query-result performance storage management caching for fast search fast top-k queries, query logging and auto-completion scoring function over many data and context criteria GUI, user guidance, personalization 34 Tasks in Web IR Related Fields Ranking Network Analysis Machine Learning Web Content Mining Routing Natural Language Processing HCI Information Science Psychology Web Usage Mining Web IR Classification Data Mining Computation al Linguistics Linguistics Sociology Multimedia analysis Question answering Clustering Databases Web IR Cultural Anthropology

7 Flickr query: August Horch Web IR research problems: an example 37 Flickr query: Koblenz 38 Motivation: multi-modal, sparse Web 2.0 data.. Co m m s Favs Tags Ca ptio n Recommender scenarios: User w Given a user, recommend photos which may be of interest. Profile w Given a user, recommend users they may like to contact. w Given a user, recommend groups they may like to join. 39 Formalizing the problem.. 40 The IR background constructing feature vectors Collaborative content sharing framework: users tags resources alice Koblenz photo 1 bob Germany photo 2 dave corner photo 3 Folksonomy cloud: user-centric resource-centric community-specific collection-specific arbitrary (e.g. groups) (e.g. favorites) Clouds of interest:.. defined analogously to tf idf 41 - favorites - groups - contact lists - comments on others resources 42 7

8 Results: user-focused favorite recommendation..spatial info? 40 Training / 50 Test favorites, 250 contrast (randomly chosen) docs Global model Personal model 43 Same coordinates, different views German corner? 44 Multi-modal Analysis of Social Media Old Balduin bridge? Steamer Goethe? Lo lev wel Co m m s Favs Uni Koblenz? Tags Time Nuclear power plant? Fortress? Ca What can you see from Ehrenbreitstein? User Profile Restaurant Ferrari? 45 ps ptio n l cia ork So tw ne o Ge Shopping malls? Gr ou 46 Course organization Lectures: w with Summer Academy: June July 2012 in A-308 Course materials: w w lecture slides & recordings available online Examination: w Oral exam at the end of the course (particular slots tba) Course organization Homework: w No mandatory assignments, classes integrated w Examples and think-about questions integrated into lectures Announcements: w Course Web Site, , twitter (#webir2013koblenz) Contact: w B-112, Wed and on appointment (sizov@uni-koblenz.de)

9 Web information retrieval: course topics Related literature w Motivation and Overview w Technical basics basics w Content processing and analysis w Link analysis & authority ranking w Advanced IR models making Web Analysis effective and efficient w SEO w Web Spam and Advertising w Web Crawling common scenarios Related literature (2) w Soumen Chakrabarti: Mining the Web: Analysis of Hypertext and Semi- Structured Data, Morgan Kaufmann, 2002 w Christopher Manning, Prabhakar Raghavan, Hinrich Schütze: Online Introduction to Information Retrieval, Cambridge University Press, 2007 w Ricardo Baeza-Yates, Berthier Ribeiro-Neto: Modern Information Retrieval: The Concepts and Technology behind Search. Addison-Wesley Professional, 2011 w Christopher D. Manning, Hinrich Schütze: Foundations of Statistical Natural Language Processing, MIT Press, w Ian H. Witten: Managing Gigabytes: Compressing and Indexing Documents and Images, Morgan Kaufmann, Additional sources important conferences on IR (see DBLP bibliography for full detail, SIGIR, ECIR, CIKM, TREC, WWW, KDD, ICDM, ICML, ECML online portals DBLP, Google Scholar, CiteSeer search engines ACM, IEEE portals Scientific mailing lists (e.g. DBWorld, AK-KDList, SIG-IRList, etc.) evaluation initiatives: Text Retrieval Conference (TREC), Cross-Language Evaluation Forum (CLEF), KDD Cup (over many years) ECML / PKDD Discovery challenge (over many years) feel free to contact.. a) lecturer, b) authors of publications, c) members of online communities and mailing lists Thank you Web search: 53 9