Machine Learning in IR. Machine Learning

Transcription

1 Machine Learning in IR CISC489/ , Lecture #21 Monday, May 4 th Ben CartereBe Machine Learning Basic idea: Given instances x in a space X and values y Learn a funcnon f(x) that maps instances to values Recall classificanon: Given: a descripnon of an instance x in X, where X is the instance space a fixed set of classes C = {c 1, c 2,, c k } Determine: the class x belongs to: f(x) in C, where f(x) is a classifica,on func,on with domain X and range C The goal is to learn funcnons that are generalizable They can correctly predict things about data never seen before 1

2 Simple Example Non classificanon example Linear regression: Data is a real dependent variable y and a vector of independent variables (or covariates, or features ) x Learn linear funcnon y i = f(x i ) = b 0 + b 1 x i1 + b 2 x i2 + FuncNon is learned by solving least squares in terms of vector b: min (yi b x i ) 2 b For a new instance x j, predict y = f(x j ) = b 0 + b 1 x j1 + b 2 x j2 + Machine Learning Tasks Regression FuncNon f(x) outputs a real number Linear regression, generalized linear models, neural networks ClassificaNon FuncNon f(x) outputs a discrete class predicnon We ve talked about this some: Naïve Bayes classifier Today: SVMs Ranking FuncNon f(x) outputs a ranking or parnal ranking RankSVM, RankNet, and more 2

3 Training and Test Data It s preby easy to fit a funcnon precisely to any given set of data Doing so does not guarantee a generalizable funcnon, though Overfiing: no ability to predict anything about new data Since the goal is to learn generalizable funcnons, we need to have some data we can test our machinelearned funcnon on Training and test splits: Given n instances (x i, y i ), use n k to learn the funcnon Use the remaining k to test it Machine Learning and IR Considerable interacnon between these fields Rocchio algorithm (60s) is a simple learning approach 80s, 90s: learning ranking algorithms based on user feedback 2000s: text categorizanon Limited by amount of training data Web query logs have generated new wave of research e.g., Learning to Rank 3

4 GeneraNve vs DiscriminaNve Two broad classes of models A genera,ve model assumes that documents were generated from some underlying model (in this case, usually a mulnnomial distribunon) and uses training data to esnmate the parameters of the model probability of belonging to a class (i.e. the relevant documents for a query) is then esnmated using Bayes Rule and the document model GeneraNve vs DiscriminaNve A discrimina,ve model esnmates the probability of belonging to a class directly from the observed features of the document based on the training data 4

5 Examples GeneraNve classifier: Recall Naïve Bayes P (y i x i )=P (x i y i )P (y i )=P (y i ) j P (x ij y i ) DiscriminaNve classifier: LogisNc regression Documents generated from class model for class y i P (y i x i )= exp(b 0 + j b jx ij ) 1 + exp(b 0 + j b jx ij ) Training a Classifier GeneraNve: Training works by esnmanng model probabilines from training data For Naïve Bayes, we esnmate P(x ij y i ) using the frequency of x ij in documents with class y i DiscriminaNve: Training works by finding feature weights b Usually by solving a minimizanon problem defined by the data For logisnc regression, the minimizanon problem is min b ( exp(b0 + j b jx ij ) i 1 + exp(b 0 + j b jx ij ) ) yi ( exp(b 0 + j b jx ij ) ) 1 yi 5

6 GeneraNve vs. DiscriminaNve All of the probabilisnc retrieval models presented so far fall into the category of genera,ve models GeneraNve models perform well with low numbers of training examples DiscriminaNve models usually have the advantage given enough training data Can also easily incorporate many features DiscriminaNve Models for IR DiscriminaNve models can be trained using explicit relevance or class judgments or click data in query logs Click data is much cheaper, more noisy e.g. Ranking Support Vector Machine (SVM) takes as input par,al rank informanon for queries parnal informanon about which documents should be ranked higher than others More on Wednesday 6

7 Support Vector Machines SVMs are a popular type of discriminanve model Their goal is to find a hyperplane separanng posinve instances from neganve instances AddiNonally, this hyperplane has maximum distance from the most difficult to classify points Maximum margin classifier If the data is linearly separable, this hyperplane is opnmal IllustraNon Linearly separable: easy classificanon problem 7

8 IllustraNon IllustraNon Line 1: 2 false posinves 6 false neganves Line 2: 6 false posinves 0 false neganves Non separable 8

9 SVM Idea There are many possible separanng hyperplanes To find the opnmal one, locate the points that are closest to the decision boundary These points are the support vectors Find the hyperplane w that has maximum distance from the support vectors Support vectors Maximize margin SVM OpNmizaNon Finding the hyperplane can be expressed as a quadranc opnmizanon problem min w,b 1 2 w w s.t. y i (w x i + b) 1 for all (x i,y i ) w b x i is the hyperplane is a bias term (intercept) is an instance (a feature vector) y i is the true class (1 or -1) 9

10 Soq Margin SVM In text classificanon problems, the data is seldom linearly separable Why? Choice of features to describe it, but also disagreement between people about which class something belongs to Modify opnmizanon problem to include slack variables These slack variables allow for some errors in placing the separanng hyperplane Soq Margin OpNmizaNon min w,b 1 2 w w + C ζ i s.t. y i (w x i + b) 1 ζ i ζ i is a slack variable for instance i ξ i ξ j 10

11 ClassificaNon with SVMs Solving the minimizanon problem produces vector w, intercept b ClassificaNon funcnon is ( ) f(x i )=sign wi x ij + b The confidence in the predicnon can be determined by the distance from the separanng hyperplane Score > t: yes Score < -t: no Else: don t know SVM EvaluaNon How do SVM and Naïve Bayes compare for a standard text classificanon task? Data: Reuters collecnon 21,578 documents (9,603 for training, 3,299 for validanon) 118 categories (documents can be in more than one) Only about 10 of the 118 are large Common categories (#train, #test) Earn (2877, 1087) Acquisitions (1650, 179) Money-fx (538, 179) Grain (433, 149) Crude (389, 189) Trade (369,119) Interest (347, 131) Ship (197, 89) Wheat (212, 71) Corn (182, 56) 11

12 Example Reuters Document <REUTERS TOPICS="YES" LEWISSPLIT="TRAIN" CGISPLIT="TRAINING-SET" OLDID="12981" NEWID="798"> <DATE> 2-MAR :51:43.42</DATE> <TOPICS><D>livestock</D><D>hog</D></TOPICS> <TITLE>AMERICAN PORK CONGRESS KICKS OFF TOMORROW</TITLE> <DATELINE> CHICAGO, March 2 - </DATELINE><BODY>The American Pork Congress kicks off tomorrow, March 3, in Indianapolis with 160 of the nations pork producers from 44 member states determining industry positions on a number of issues, according to the National Pork Producers Council, NPPC. Delegates to the three day Congress will be considering 26 resolutions concerning various issues, including the future direction of farm policy and the tax law as it applies to the agriculture sector. The delegates will also debate whether to endorse concepts of a national PRV (pseudorabies virus) control and eradication program, the NPPC said. A large trade show, in conjunction with the congress, will feature the latest in technology in all areas of the industry, the NPPC added. Reuter </BODY></TEXT></REUTERS> ConNngency table: EvaluaNng a Classifier In class Not in class Predicted in class A B A+B Predicted not in class C D C+D A, B, C, and D are counts precision = recall = A A + B A A + C A+C B+D N=A+B+C+D accuracy = F 1 = A + D A + B + C + D 2PR P + R What to do when there are more than two classes? Macro averaging: calculate evaluanon measure for each class, then average over classes Micro averaging: add all class connngency tables together, then calculate measures 12

13 Non Linear SVMs We have only discussed the linear case, but SVMs are not restricted to that case If the data is not linearly separable, map it into a higherdimensional space where it might become linearly separable Φ: x φ(x) 13

14 Feature SelecNon Recall that Naïve Bayes classificanon required some feature selecnon to perform well DiscriminaNve models are generally more robust to noninformanve features They will simply end up with near zero weights In pracnce, feature selecnon is snll useful to reduce the number of weights that must be esnmated 14