CSE 5243 INTRO. TO DATA MINING

Transcription

1 CSE 5243 INTRO. TO DATA MINING Graph Data & Introduction to Information Retrieval Huan Sun, Ohio State University 11/21/2017 Slides adapted from Prof. Srinivasan

2 2 Chapter 4 Graph Data: athuploads?action=downloadman&upname=book pdf, GRAPH BASICS AND A GENTLE INTRODUCTION TO PAGERANK Slides adapted from Prof. Srinivasan

3 3 Background Besides the keywords, what other evidence can one use to rate the importance of a webpage? Solution: Use the hyperlink structure E.g. a webpage linked by many webpages is probably important. but this method is not global (comprehensive). PageRank is developed by Larry Page in 1998.

4 4 Idea A graph representing WWW Node: webpage Directed edge: hyperlink A user randomly clicks the hyperlink to surf WWW. The probability a user stop in a particular webpage is the PageRank value. A node that is linked by many nodes with high PageRank value receives a high rank itself; If there are no links to a node, then there is no support for that page.

5 5 Formal Formulation

6 6 Formal Formulation

7 7 Iterative Computation

8 Example 1 =the transpose of A (adjacency matrix) 8 PageRank Calculation: first iteration

9 Example 1 9 PageRank Calculation: second iteration

10 Example 1 10 Convergence after some iterations

11 A simple version u: a webpage B u : the set of u s backlinks R( u) = v B u N v : the number of forward links of page v R( v) N v Initially, R(u) is 1/N for every webpage Iteratively update each webpage s PR value until convergence. 11

12 A little more advanced version Adding a damping factor d Imagine that a surfer would stop clicking a hyperlink with probability 1-d (1 d ) R( v) R( u) = N 1 + d v B u N v R(u) is at least (1-d)/(N-1) N is the total number of nodes. 12

13 Other applications Social network (Facebook, Twitter, etc) Node: Person; Edge: Follower / Followee / Friend Higher PR value: Celebrity Citation network Node: Paper; Edge: Citation Higher PR values: Important Papers. Protein-protein interaction network Node: Protein; Edge: Two proteins bind together Higher PR values: Essential proteins. 13

14 SEARCH ENGINES INFORMATION RETRIEVAL IN PRACTICE BOOK: SLIDES: All slides Addison Wesley, 2008 Slides adapted from Prof. W. Bruce

15 15 Search Engines and Information Retrieval All slides Addison Wesley, 2008 Information Retrieval in Practice

16 Search and Information Retrieval Search on the Web is a daily activity for many people throughout the world Search and communication are most popular uses of the computer Applications involving search are everywhere The field of computer science that is most involved with R&D for search is information retrieval (IR) 16

17 17 Information Retrieval Information retrieval is a field concerned with the structure, analysis, organization, storage, searching, and retrieval of information. (Salton, 1968) General definition that can be applied to many types of information and search applications Primary focus of IR since the 50s has been on text and documents

18 What is a Document? Examples: web pages, , books, news stories, scholarly papers, text messages, Word, Powerpoint, PDF, forum postings, patents, IM sessions, etc. Common properties Significant text content Some structure (e.g., title, author, date for papers; subject, sender, destination for ) 18

19 Documents vs. Database Records Database records (or tuples in relational databases) are typically made up of well-defined fields (or attributes) e.g., bank records with account numbers, balances, names, addresses, social security numbers, dates of birth, etc. Easy to compare fields with well-defined semantics to queries in order to find matches Text is more difficult 19

20 Documents vs. Records Example bank database query Find records with balance > $50,000 in branches located in Amherst, MA. Matches easily found by comparison with field values of records Example search engine query bank scandals in western mass This text must be compared to the text of entire news stories 20

21 Comparing Text Comparing the query text to the document text and determining what is a good match is the core issue of information retrieval Exact matching of words is not enough Many different ways to write the same thing in a natural language like English e.g., does a news story containing the text bank director in Amherst steals funds match the query? Some stories will be better matches than others 21

22 Dimensions of IR IR is more than just text, and more than just web search although these are central People doing IR work with different media, different types of search applications, and different tasks 22

23 Other Media New applications increasingly involve new media e.g., video, photos, music, speech Like text, content is difficult to describe and compare text may be used to represent them (e.g. tags) IR approaches to search and evaluation are appropriate 23

24 24 Dimensions of IR Content Applications Tasks Text Web search Ad hoc search Images Vertical search Filtering Video Enterprise search Classification Scanned docs Desktop search Question answering Audio Forum search Music P2P search Literature search

25 IR Tasks Ad-hoc search Find relevant documents for an arbitrary text query Filtering Identify relevant user profiles for a new document Classification Identify relevant labels for documents Question answering Give a specific answer to a question 25

26 Big Issues in IR Relevance What is it? Simple (and simplistic) definition: A relevant document contains the information that a person was looking for when they submitted a query to the search engine Many factors influence a person s decision about what is relevant: e.g., task, context, novelty, style Topical relevance (same topic) vs. user relevance (everything else) 26

27 27 Big Issues in IR Relevance Retrieval models define a view of relevance Ranking algorithms used in search engines are based on retrieval models Most models describe statistical properties of text rather than linguistic i.e. counting simple text features such as words instead of parsing and analyzing the sentences Statistical approach to text processing started with Luhn in the 50s Linguistic features can be part of a statistical model

28 Big Issues in IR Evaluation Experimental procedures and measures for comparing system output with user expectations Originated in Cranfield experiments in the 60s IR evaluation methods now used in many fields Typically use test collection of documents, queries, and relevance judgments Most commonly used are TREC collections Recall and precision are two examples of effectiveness measures 28

29 Big Issues in IR Users and Information Needs Search evaluation is user-centered Keyword queries are often poor descriptions of actual information needs Interaction and context are important for understanding user intent Query refinement techniques such as query expansion, query suggestion, relevance feedback improve ranking 29

30 IR and Search Engines A search engine is the practical application of information retrieval techniques to large scale text collections Web search engines are best-known examples, but many others Open source search engines are important for research and development e.g., Lucene, Lemur/Indri, Galago Big issues include main IR issues but also some others 30

31 IR and Search Engines Information Retrieval Search Engines Relevance -Effective ranking Evaluation -Testing and measuring Information needs -User interaction Performance -Efficient search and indexing Incorporating new data -Coverage and freshness Scalability -Growing with data and users Adaptability -Tuning for applications Specific problems -e.g. Spam 31

32 Search Engine Issues Performance Measuring and improving the efficiency of search e.g., reducing response time, increasing query throughput, increasing indexing speed Indexes are data structures designed to improve search efficiency designing and implementing them are major issues for search engines 32

33 Search Engine Issues Dynamic data The collection for most real applications is constantly changing in terms of updates, additions, deletions e.g., web pages Acquiring or crawling the documents is a major task Typical measures are coverage (how much has been indexed) and freshness (how recently was it indexed) Updating the indexes while processing queries is also a design issue 33

34 Search Engine Issues Scalability Making everything work with millions of users every day, and many terabytes of documents Distributed processing is essential Adaptability Changing and tuning search engine components such as ranking algorithm, indexing strategy, interface for different applications 34

35 35 Architecture of a Search Engine All slides Addison Wesley, 2008 Information Retrieval in Practice

36 Search Engine Architecture A software architecture consists of software components, the interfaces provided by those components, and the relationships between them describes a system at a particular level of abstraction Architecture of a search engine determined by 2 requirements effectiveness (quality of results) and efficiency (response time and throughput) 36

37 37 Indexing Process

38 Indexing Process Text acquisition identifies and stores documents for indexing Text transformation transforms documents into index terms or features Index creation takes index terms and creates data structures (indexes) to support fast searching 38

39 39 Query Process

40 Query Process User interaction supports creation and refinement of query, display of results Ranking uses query and indexes to generate ranked list of documents Evaluation monitors and measures effectiveness and efficiency (primarily offline) 40

41 Details: Text Acquisition Crawler Identifies and acquires documents for search engine Many types web, enterprise, desktop Web crawlers follow links to find documents Must efficiently find huge numbers of web pages (coverage) and keep them up-to-date (freshness) Single site crawlers for site search Topical or focused crawlers for vertical search Document crawlers for enterprise and desktop search Follow links and scan directories 41

42 Text Acquisition Feeds Real-time streams of documents e.g., web feeds for news, blogs, video, radio, tv RSS is common standard RSS reader can provide new XML documents to search engine Conversion Convert variety of documents into a consistent text plus metadata format e.g. HTML, XML, Word, PDF, etc. XML Convert text encoding for different languages Using a Unicode standard like UTF-8 42

43 Text Acquisition Document data store Stores text, metadata, and other related content for documents Metadata is information about document such as type and creation date Other content includes links, anchor text Provides fast access to document contents for search engine components e.g. result list generation Could use relational database system More typically, a simpler, more efficient storage system is used due to huge numbers of documents 43

44 44 Text Transformation Parser Processing the sequence of text tokens in the document to recognize structural elements e.g., titles, links, headings, etc. Tokenizer recognizes words in the text must consider issues like capitalization, hyphens, apostrophes, nonalpha characters, separators Markup languages such as HTML, XML often used to specify structure Tags used to specify document elements E.g., <h2> Overview </h2> Document parser uses syntax of markup language (or other formatting) to identify structure

45 45 Text Transformation Stopping Remove common words e.g., and, or, the, in Some impact on efficiency and effectiveness Can be a problem for some queries Stemming Group words derived from a common stem e.g., computer, computers, computing, compute Usually effective, but not for all queries Benefits vary for different languages

46 Text Transformation Link Analysis Makes use of links and anchor text in web pages Link analysis identifies popularity and community information e.g., PageRank Anchor text can significantly enhance the representation of pages pointed to by links Significant impact on web search Less importance in other applications 46

47 Text Transformation Information Extraction Identify classes of index terms that are important for some applications e.g., named entity recognizers identify classes such as people, locations, companies, dates, etc. Classifier Identifies class-related metadata for documents i.e., assigns labels to documents e.g., topics, reading levels, sentiment, genre Use depends on application 47

48 Index Creation Document Statistics Gathers counts and positions of words and other features Used in ranking algorithm Weighting Computes weights for index terms Used in ranking algorithm e.g., tf.idf weight Combination of term frequency in document and inverse document frequency in the collection 48

49 Index Creation Inversion Core of indexing process Converts document-term information to term-document for indexing Difficult for very large numbers of documents Format of inverted file is designed for fast query processing Must also handle updates Compression used for efficiency 49

50 Index Creation Index Distribution Distributes indexes across multiple computers and/or multiple sites Essential for fast query processing with large numbers of documents Many variations Document distribution, term distribution, replication P2P and distributed IR involve search across multiple sites 50

51 User Interaction Query input Provides interface and parser for query language Most web queries are very simple, other applications may use forms Query language used to describe more complex queries and results of query transformation e.g., Boolean queries, Indri and Galago query languages similar to SQL language used in database applications IR query languages also allow content and structure specifications, but focus on content 51

52 User Interaction Query transformation Improves initial query, both before and after initial search Includes text transformation techniques used for documents Spell checking and query suggestion provide alternatives to original query Query expansion and relevance feedback modify the original query with additional terms 52

53 User Interaction Results output Constructs the display of ranked documents for a query Generates snippets to show how queries match documents Highlights important words and passages Retrieves appropriate advertising in many applications May provide clustering and other visualization tools 53

54 Ranking Scoring Calculates scores for documents using a ranking algorithm Core component of search engine Basic form of score is q i d i q i and d i are query and document term weights for term i Many variations of ranking algorithms and retrieval models 54

55 Ranking Performance optimization Designing ranking algorithms for efficient processing Term-at-a time vs. document-at-a-time processing Safe vs. unsafe optimizations Distribution Processing queries in a distributed environment Query broker distributes queries and assembles results Caching is a form of distributed searching 55

56 Evaluation Logging Logging user queries and interaction is crucial for improving search effectiveness and efficiency Query logs and clickthrough data used for query suggestion, spell checking, query caching, ranking, advertising search, and other components Ranking analysis Measuring and tuning ranking effectiveness Performance analysis Measuring and tuning system efficiency 56

57 57 How Does It Really Work? The course* explains these components of a search engine in more detail Often many possible approaches and techniques for a given component Focus is on the most important alternatives i.e., explain a small number of approaches in detail rather than many approaches Importance based on research results and use in actual search engines Alternatives described in references *

58 RETRIEVAL MODELS All slides Addison Wesley, 2008 Information Retrieval in Practice

59 Retrieval Models Provide a mathematical framework for defining the search process includes explanation of assumptions basis of many ranking algorithms can be implicit Progress in retrieval models has corresponded with improvements in effectiveness Theories about relevance 59

60 Relevance Complex concept that has been studied for some time Many factors to consider People often disagree when making relevance judgments Retrieval models make various assumptions about relevance to simplify problem e.g., topical vs. user relevance e.g., binary vs. multi-valued relevance 60

61 Retrieval Model Overview Older models Boolean retrieval Vector Space model Probabilistic Models BM25 Language models Combining evidence Inference networks Learning to Rank 61

62 Boolean Retrieval Two possible outcomes for query processing TRUE and FALSE exact-match retrieval simplest form of ranking Query usually specified using Boolean operators AND, OR, NOT proximity operators also used 62

63 Boolean Retrieval Advantages Results are predictable, relatively easy to explain Many different features can be incorporated Efficient processing since many documents can be eliminated from search Disadvantages Effectiveness depends entirely on user Simple queries usually don t work well Complex queries are difficult 63

64 Searching by Numbers Sequence of queries driven by number of retrieved documents e.g. lincoln search of news articles president AND lincoln president AND lincoln AND NOT (automobile OR car) president AND lincoln AND biography AND life AND birthplace AND gettysburg AND NOT (automobile OR car) president AND lincoln AND (biography OR life OR birthplace OR gettysburg) AND NOT (automobile OR car) 64

65 Vector Space Model Documents and query represented by a vector of term weights Collection represented by a matrix of term weights 65

66 66 Vector Space Model

67 67 Vector Space Model 3-d pictures useful, but can be misleading for high-dimensional space

68 Vector Space Model Documents ranked by distance between points representing query and documents Similarity measure more common than a distance or dissimilarity measure e.g. Cosine correlation 68

69 Similarity Calculation Consider two documents D 1, D 2 and a query Q D 1 = (0.5, 0.8, 0.3), D 2 = (0.9, 0.4, 0.2), Q = (1.5, 1.0, 0) 69

70 Term Weights tf.idf weight Term frequency weight measures importance in document: Inverse document frequency measures importance in collection: Some heuristic modifications 70

71 Relevance Feedback Rocchio algorithm Optimal query Maximizes the difference between the average vector representing the relevant documents and the average vector representing the non-relevant documents Modifies query according to α, β, and γ are parameters Typical values 8, 16, 4 71

72 Vector Space Model Advantages Simple computational framework for ranking Any similarity measure or term weighting scheme could be used Disadvantages Assumption of term independence No predictions about techniques for effective ranking 72

73 Probability Ranking Principle Robertson (1977) If a reference retrieval system s response to each request is a ranking of the documents in the collection in order of decreasing probability of relevance to the user who submitted the request, where the probabilities are estimated as accurately as possible on the basis of whatever data have been made available to the system for this purpose, the overall effectiveness of the system to its user will be the best that is obtainable on the basis of those data. 73

74 74 IR as Classification

75 Bayes Classifier Bayes Decision Rule A document D is relevant if P(R D) > P(NR D) Estimating probabilities use Bayes Rule classify a document as relevant if lhs is likelihood ratio 75

76 Estimating P(D R) Assume independence Binary independence model document represented by a vector of binary features indicating term occurrence (or non-occurrence) p i is probability that term i occurs (i.e., has value 1) in relevant document, s i is probability of occurrence in non-relevant document 76

77 77 Binary Independence Model

78 Binary Independence Model Scoring function is Query provides information about relevant documents If we assume p i constant, s i approximated by entire collection, get idflike weight 78

79 79 Contingency Table Gives scoring function:

80 BM25 Popular and effective ranking algorithm based on binary independence model adds document and query term weights k1, k2 and K are parameters whose values are set empirically dl is doc length Typical TREC value for k1 is 1.2, k2 varies from 0 to 1000, b =

81 81 BM25 Example Query with two terms, president lincoln, (qf = 1) No relevance information (r and R are zero) N = 500,000 documents president occurs in 40,000 documents (n 1 = 40, 000) lincoln occurs in 300 documents (n 2 = 300) president occurs 15 times in doc (f 1 = 15) lincoln occurs 25 times (f 2 = 25) document length is 90% of the average length (dl/avdl =.9) k 1 = 1.2, b = 0.75, and k 2 = 100 K = 1.2 ( ) = 1.11

82 82 BM25 Example

83 83 BM25 Example Effect of term frequencies

84 Language Model Unigram language model probability distribution over the words in a language generation of text consists of pulling words out of a bucket according to the probability distribution and replacing them N-gram language model some applications use bigram and trigram language models where probabilities depend on previous words 84

85 Language Model A topic in a document or query can be represented as a language model i.e., words that tend to occur often when discussing a topic will have high probabilities in the corresponding language model Multinomial distribution over words text is modeled as a finite sequence of words, where there are t possible words at each point in the sequence commonly used, but not only possibility doesn t model burstiness 85

86 LMs for Retrieval 3 possibilities: probability of generating the query text from a document language model probability of generating the document text from a query language model comparing the language models representing the query and document topics Models of topical relevance 86

87 Query-Likelihood Model Rank documents by the probability that the query could be generated by the document model (i.e. same topic) Given query, start with P(D Q) Using Bayes Rule Assuming prior is uniform, unigram model 87

88 Estimating Probabilities Obvious estimate for unigram probabilities is Maximum likelihood estimate makes the observed value of fqi;d most likely If query words are missing from document, score will be zero Missing 1 out of 4 query words same as missing 3 out of 4 88

89 Smoothing Document texts are a sample from the language model Missing words should not have zero probability of occurring Smoothing is a technique for estimating probabilities for missing (or unseen) words lower (or discount) the probability estimates for words that are seen in the document text assign that left-over probability to the estimates for the words that are not seen in the text 89

90 Estimating Probabilities Estimate for unseen words is α D P(q i C) P(q i C) is the probability for query word i in the collection language model for collection C (background probability) α D is a parameter Estimate for words that occur is (1 α D ) P(q i D) + α D P(q i C) Different forms of estimation come from different α D 90

91 Jelinek-Mercer Smoothing α D is a constant, λ Gives estimate of Ranking score Use logs for convenience accuracy problems multiplying small numbers 91

92 Where is tf.idf Weight? 92 - proportional to the term frequency, inversely proportional to the collection frequency

93 93 Dirichlet Smoothing α D depends on document length Gives probability estimation of and document score

94 Query Likelihood Example For the term president f qi,d = 15, c qi = 160,000 For the term lincoln f qi,d = 25, c qi = 2,400 number of word occurrences in the document d is assumed to be 1,800 number of word occurrences in the collection is ,000 documents times an average of 2,000 words μ = 2,000 94

95 95 Query Likelihood Example Negative number because summing logs of small numbers

96 96 Query Likelihood Example

97 Relevance Models Relevance model language model representing information need query and relevant documents are samples from this model P(D R) - probability of generating the text in a document given a relevance model document likelihood model less effective than query likelihood due to difficulties comparing across documents of different lengths 97

98 Pseudo-Relevance Feedback Estimate relevance model from query and top-ranked documents Rank documents by similarity of document model to relevance model Kullback-Leibler divergence (KL-divergence) is a well-known measure of the difference between two probability distributions 98

99 99 KL-Divergence Given the true probability distribution P and another distribution Q that is an approximation to P, Use negative KL-divergence for ranking, and assume relevance model R is the true distribution (not symmetric),

100 100 KL-Divergence Given a simple maximum likelihood estimate for P(w R), based on the frequency in the query text, ranking score is rank-equivalent to query likelihood score Query likelihood model is a special case of retrieval based on relevance model

101 101 Estimating the Relevance Model Probability of pulling a word w out of the bucket representing the relevance model depends on the n query words we have just pulled out By definition

102 102 Estimating the Relevance Model Joint probability is Assume Gives

103 Estimating the Relevance Model P(D) usually assumed to be uniform P(w, q1... qn) is simply a weighted average of the language model probabilities for w in a set of documents, where the weights are the query likelihood scores for those documents Formal model for pseudo-relevance feedback query expansion technique 103

104 104 Pseudo-Feedback Algorithm

105 105 Example from Top 10 Docs

106 106 Example from Top 50 Docs

107 Combining Evidence Effective retrieval requires the combination of many pieces of evidence about a document s potential relevance have focused on simple word-based evidence many other types of evidence structure, PageRank, metadata, even scores from different models Inference network model is one approach to combining evidence uses Bayesian network formalism 107

108 108 Inference Network

109 Inference Network Document node (D) corresponds to the event that a document is observed Representation nodes (r i ) are document features (evidence) Probabilities associated with those features are based on language models θ estimated using the parameters μ one language model for each significant document structure r i nodes can represent proximity features, or other types of evidence (e.g. date) 109

110 Inference Network Query nodes (q i ) are used to combine evidence from representation nodes and other query nodes represent the occurrence of more complex evidence and document features a number of combination operators are available Information need node (I) is a special query node that combines all of the evidence from the other query nodes network computes P(I D, μ) 110

111 111 Example: AND Combination a and b are parent nodes for q

112 112 Example: AND Combination Combination must consider all possible states of parents Some combinations can be computed efficiently

113 113 Inference Network Operators

114 114 Backup slides

115 115 Galago Query Language A document is viewed as a sequence of text that may contain arbitrary tags A single context is generated for each unique tag name An extent is a sequence of text that appears within a single begin/end tag pair of the same type as the context

116 116 Galago Query Language

117 117 Galago Query Language TexPoint Display

118 118 Galago Query Language

119 119 Galago Query Language

120 120 Galago Query Language

121 121 Galago Query Language

122 122 Galago Query Language

123 123 Galago Query Language

124 Web Search Most important, but not only, search application Major differences to TREC news Size of collection Connections between documents Range of document types Importance of spam Volume of queries Range of query types 124

125 Search Taxonomy Informational Finding information about some topic which may be on one or more web pages Topical search Navigational finding a particular web page that the user has either seen before or is assumed to exist Transactional finding a site where a task such as shopping or downloading music can be performed 125

126 Web Search For effective navigational and transactional search, need to combine features that reflect user relevance Commercial web search engines combine evidence from hundreds of features to generate a ranking score for a web page page content, page metadata, anchor text, links (e.g., PageRank), and user behavior (click logs) page metadata e.g., age, how often it is updated, the URL of the page, the domain name of its site, and the amount of text content 126

127 Search Engine Optimization SEO: understanding the relative importance of features used in search and how they can be manipulated to obtain better search rankings for a web page e.g., improve the text used in the title tag, improve the text in heading tags, make sure that the domain name and URL contain important keywords, and try to improve the anchor text and link structure Some of these techniques are regarded as not appropriate by search engine companies 127

128 Web Search In TREC evaluations, most effective features for navigational search are: text in the title, body, and heading (h1, h2, h3, and h4) parts of the document, the anchor text of all links pointing to the document, the PageRank number, and the inlink count Given size of Web, many pages will contain all query terms Ranking algorithm focuses on discriminating between these pages Word proximity is important 128

129 129 Term Proximity Many models have been developed N-grams are commonly used in commercial web search Dependence model based on inference net has been effective in TREC - e.g.

130 130 Example Web Query

131 Machine Learning and IR Considerable interaction between these fields Rocchio algorithm (60s) is a simple learning approach 80s, 90s: learning ranking algorithms based on user feedback 2000s: text categorization Limited by amount of training data Web query logs have generated new wave of research e.g., Learning to Rank 131

132 Generative vs. Discriminative All of the probabilistic retrieval models presented so far fall into the category of generative models A generative model assumes that documents were generated from some underlying model (in this case, usually a multinomial distribution) and uses training data to estimate the parameters of the model probability of belonging to a class (i.e. the relevant documents for a query) is then estimated using Bayes Rule and the document model 132

133 Generative vs. Discriminative A discriminative model estimates the probability of belonging to a class directly from the observed features of the document based on the training data Generative models perform well with low numbers of training examples Discriminative models usually have the advantage given enough training data Can also easily incorporate many features 133

134 Discriminative Models for IR Discriminative models can be trained using explicit relevance judgments or click data in query logs Click data is much cheaper, more noisy e.g. Ranking Support Vector Machine (SVM) takes as input partial rank information for queries partial information about which documents should be ranked higher than others 134

135 Ranking SVM Training data is r is partial rank information if document dashould be ranked higher than db, then (da, db) ri partial rank information comes from relevance judgments (allows multiple levels of relevance) or click data e.g., d1, d2 and d3 are the documents in the first, second and third rank of the search output, only d3 clicked on (d3, d1) and (d3, d2) will be in desired ranking for this query 135

136 Ranking SVM Learning a linear ranking function where w is a weight vector that is adjusted by learning d a is the vector representation of the features of document non-linear functions also possible Weights represent importance of features learned using training data e.g., 136

137 137 Ranking SVM Learn w that satisfies as many of the following conditions as possible: Can be formulated as an optimization problem

138 138 Ranking SVM ξ, known as a slack variable, allows for misclassification of difficult or noisy training examples, and C is a parameter that is used to prevent overfitting

139 Ranking SVM Software available to do optimization Each pair of documents in our training data can be represented by the vector: Score for this pair is: SVM classifier will find a w that makes the smallest score as large as possible make the differences in scores as large as possible for the pairs of documents that are hardest to rank 139

140 Topic Models Improved representations of documents can also be viewed as improved smoothing techniques improve estimates for words that are related to the topic(s) of the document instead of just using background probabilities Approaches Latent Semantic Indexing (LSI) Probabilistic Latent Semantic Indexing (plsi) Latent Dirichlet Allocation (LDA) 140

141 141 LDA Model document as being generated from a mixture of topics

142 142 LDA Gives language model probabilities Used to smooth the document representation by mixing them with the query likelihood probability as follows:

143 LDA If the LDA probabilities are used directly as the document representation, the effectiveness will be significantly reduced because the features are too smoothed e.g., in typical TREC experiment, only 400 topics used for the entire collection generating LDA topics is expensive When used for smoothing, effectiveness is improved 143

144 144 LDA Example Top words from 4 LDA topics from TREC news

145 Summary Best retrieval model depends on application and data available Evaluation corpus (or test collection), training data, and user data are all critical resources Open source search engines can be used to find effective ranking algorithms Galago query language makes this particularly easy Language resources (e.g., thesaurus) can make a big difference 145