Homework : Data Mining SOLUTIONS

Transcription

1 Homework : Data Mining SOLUTIONS 1. (a) What is the bag-of-words representation of the sentence to be or not to be? Answer: A vector with one component for each word in our dictionary, all of them zero except for the following: be not or to This is the format given by table(c("to","be","or","not","to","be")) (b) Suppose we search for the above sentence via the keyword be. What is the bag-of-words representation for this query, and what is the Euclidean distance from the sentence? Answer: A vector whose only non-zero component is that for be, where the count is 1. The Euclidean distance is (2 1)2 + (1 0) 2 + (1 0) + (0 2) 2 = 7 (c) Describe how weighting words by inverse-document-frequency (IDF) should help when making a Web query for The Principles of Data Mining. Answer: It keeps us from wasting time on words like the and of, and emphasize the less-common, more-informative words principles, data and mining ; something titled Data Mining Principles is a good match. (d) Describe a simple text search that could not be carried out effectively using a bag-of-words representation (no matter what distance measure is used). Simple means no actual understanding of English is required. Answer: There are many; but a search for the exact phrase to be or not to be is impossible. 1

2 2. (a) What is the Euclidean distance between each of the vectors (1, 0, 0), (1, 4, 5), and (10, 0, 0)? Answer: The distance between the first and second vectors is ; between the first and third is 9; and between the second and third is (b) Divide each vector by its sum. How do the relative distances change? Answer: The first and third vectors become the closest pair. (c) Divide each vector by its Euclidean length. How do the relative distances change? Answer: The first and third vectors again become the closest pair. (d) Suppose we re using the bag-of-words representation for similarity searching with a Euclidean metric. Describe how the previous parts of the question illustrate a potential problem if we do not normalize for document length. Answer: Documents with the same distribution of words but different sizes will appear to be very far apart. (e) Consider the conventional searching scheme where the user picks a set of keywords and the system returns all documents containing those keywords. Describe how the previous parts of the question illustrate a potential problem with this type of search. Answer: The second vector contains the first keyword, but it s a small part of it, and return it because it s a match will lower the precision. 2

3 3. (a) Create document vectors for each of the stories in the music and art folders. Give the commands you used. Answer: I ll use the read.directory function, so I ll comment it first (in somewhat more detail than you would need to). # Read in all the files in a directory as vectors of character strings # Inputs: directory name (dirname), flag for verbose output (verbose) # Calls: read.doc() in 01.R # Output: List of vectors, each containing the words of one file in order read.directory <- function(dirname,verbose=false) { stories = list() # Make a list to store the story-vectors in filenames = dir(dirname,full.names=true) # List the files in the directory # Using the full.names option means we can pass the results to other # commands without worrying about the current working directory, etc. for (i in 1:length(filenames)) { # For each file, if(verbose) { print(filenames[i]) # Print the filename before we try anything with it (if we re verbose) stories[[i]] = read.doc(filenames[i]) # Read in file, stick it in the list # read.doc only works on XML files, should really check that first! return(stories) # Return the list Now I can read in the stories: > library(xml) > source("~/teaching/350/hw/01/01.r") > music <- read.directory("~/teaching/350/hw/01/nyt_corpus/music") > art <- read.directory("~/teaching/350/hw/01/nyt_corpus/art") > length(music) [1] 45 > length(art) [1] 57 > is.list(music) [1] TRUE > is.list(art) [1] TRUE > length(music[[1]]) [1] 413 > is.vector(music[[1]]) [1] TRUE (b) What command would you use to extract the 37 th word of story number in art? (That word is experiencing.) Give a command 3

4 to count the number of times the word the appears in that story. (There are at least two ways to do this. The correct answer is 103.) Answer: The list art contains a vector which has story number , but which? I can either count on a directory listing, or I can have R count for me: > which(dir("~/teaching/350/hw/01/nyt_corpus/art")==" xml") [1] 48 (Note that here I don t use the full.names option to dir(), since I don t want the full path to the files.) So the story I want is art[[48]]. The 37 th word is > art[[48]][37] [1] "experiencing" One way to see how often the appears is to do this: > sum(art[[48]] == "the") [1] 103 The expression inside sum() creates a logical vector, which is TRUE at the words which are equal to the and FALSE elsewhere. The sum of a logical vector is the number of TRUE components, i.e., number of instances of the. Alternately, use table: > table(art[[48]])["the"] the 103 Applied to a vector, table creates an array 1 where the columns are the distinct values from the vector, and entries are how often those values occur. The values themselves become the column names, so we can check how often the occurs without having to know where it comes in the vocabulary list (place 633, as it happens). R will print the column name when you access the column, but it will do arithmetic on it as usual. > table(art[[48]])["the"] + 17 the 120 (c) Give the commands you would use to construct a bag-of-words dataframe from the document vectors for the art and music stories. Hint: consider lapply. Answer: I ll take the hint. > art.bow <- lapply(art,table) > music.bow <- lapply(music,table) > is.list(music.bow) 1 R Pedantry: Really, an object of class table, which inherits from the class array. 4

5 [1] TRUE > length(music.bow) [1] 45 lapply is one of the most useful functions in R: it takes two arguments, a data structure (vector, list, array,...) and a function, and returns a list which it gets by applying the function to each element of the data structure. (It has a variant, sapply, which will try to automatically simplify the result down to a vector, but that s not what s wanted here.) The table command we ve already discussed. The last two commands just check that things are working as they ought to. > nyt.frame <- make.bow.frame(c(art.bow,music.bow)) > dim(nyt.frame) [1] The number of rows should equal the total number of stories, and = 102. (d) Create distance matrices from this data frame for (a) the straight Euclidean distance, (b) the distance with word-count normalization and (c) the distance with vector-length scaling, and then for all three again with inverse-document-frequency weighting. Give the commands you use. Answer: The distances command can be used for all of these, with the right scalings and normalization. There are already functions in 01.R to do these. dist.plain = distances(nyt.frame) dist.wordcount = distances(div.by.sum(nyt.frame)) dist.euclen = distances(div.by.euc.length(nyt.frame)) nyt.frame.idf = idf.weight(nyt.frame) dist.idf.plain = distances(nyt.frame.idf) dist.idf.wordcount = distances(div.by.sum(nyt.frame.idf)) dist.idf.euclen = distances(div.by.euc.length(nyt.frame.idf)) Quick checks that these are acting sensibly: > dim(dist.idf.euclen) [1] > dim(dist.plain) [1] > round(dist.plain[1:5,1:5],2) [,1] [,2] [,3] [,4] [,5] [1,] [2,] [3,] [4,] [5,]

6 So I get matrices which are symmetric and have zeroes down the diagonal looks good. (e) For each of the six different difference measures, what is the average distance between stories in the same category and between stories in different categories? (Include the R command you use to compute this don t do it by hand!) Answer: There are multiple ways to do this. The simplest is to realize that, in this case, the first 57 stories are all art, and the last 45 are all music. So if d is a distance matrix, the within-category entries are d[1:57,1:57] and d[58:102,58:102], and the betweencategory entries are d[1:57,58:102]. So mean(c(d[1:57,1:57],d[58:102,58:102])) would give the average distance between stories in the same category, similarly mean(d[58:102,1:57]) for the between-category average. Of course, that trick relies on the categories forming two solid blocks. Here s another which isn t quite so fragile. First, we create a vector which gives the class labels. class.labels = c(rep("art",57),rep("music",45)) Now create a logical matrix which says whether or not two documents belong to different classes. are.different = outer(class.labels,class.labels,"!=") The outer() function takes three arguments: two data-structures and another function. (Here the function is!=, which I put in quotes so that R realizes I m naming a function, and not asking it to evaluate an expression.) It returns a matrix which it gets from applying the function to each pair of components from its first two arguments. Here those first two arguments are vectors of length 102, so what it gives back is a matrix, where are.different[i,j] shows whether class.label[i]!= class.label[j]. In other words, it s TRUE if documents i and j belong to different classes. And a logical array picks out elements from another array: mean(d[are.different]) is the average distance between classes. To average within classes, mean(d[!are.different]) Not only does this work if the classes are intermingled (we just have to get the class.labels vector right), we can also use this to not include the distance from a document to itself in the within-class average: are.same =!are.different diag(are.same) = FALSE mean(d[are.same]) 6

7 Mean distance without IDF with IDF plain sum-normed length-normed plain sum-normed length-normed within-class, with self within-class, without self between-class This makes for a fairer comparison of the in-class and cross-class averages, but there s no penalty on this homework for not doing this. 7

8 (f) Create multidimensional scaling plots for the different distances, and describe what you see. Include the code you used, the plots, and explanations for the code. Answer: I ll use the basic command cmdscale(), which returns a matrix whose columns give the new coordinates. In fact, I wrote a function which calls cmdscale(), and then plots the points with different colors and/or shapes for the different classes. This exploits the useful graphics trick that if you can control the color and shape of individual points by making the col and pch arguments to the plotting command into vectors. # Multidimensional-scaling plot for labeled points, with color and/or shape # indicating classes # Inputs: Distance matrix (distances) # vector of class labels (labels) # vector of class color names (class.colors) # defaults to red and blue # vector of class point shapes (class.shapes) --- see?pch # defaults to circles and squares # optional plotting parameters # Calls: cmdscale() # Output: invisibly returns matrix of coordinates for the MDS image points my.mds <- function(distances, labels,class.colors=c("red","blue"), class.shapes=c(21,22),...) { # Should really check that: # distances is a matrix of non-negative numbers, size = (length of labels)^2 # class.colors and class.shapes have reasonable lengths label.values = unique(labels) # What are the different labels? names(class.colors) = label.values # Allows access by class value as opposed names(class.shapes) = label.values # to just position num.labels = length(label.values) # Re-cycle the colors if need be # Useful if we want to have only one color but multiple shapes if (length(class.colors) < num.labels) { class.colors = rep(class.colors,length.out=num.labels) # Ditto shapes if (length(class.shapes) < num.labels) { class.shapes = rep(class.shapes,length.out=num.labels) cols.vec = class.colors[labels] # Vector giving class colors in order shapes.vec = class.shapes[labels] # Ditto shapes mds.coords = cmdscale(distances) # Get the MDS coordinates plot(mds.coords,col=cols.vec,pch=shapes.vec,...) # Actual plotting return(invisible(mds.coords)) 8

9 No normalization, no weighting Word-count normalization, no weighting Euclidean length normalization, no weighting mds.coords[,2] mds.coords[,2] mds.coords[,2] mds.coords[,1] mds.coords[,1] mds.coords[,1] No normalization, IDF weighting Word-count normalization, IDF weighting Euclidean length normalization, IDF weighting mds.coords[,2] mds.coords[,2] mds.coords[,2] mds.coords[,1] mds.coords[,1] mds.coords[,1] Figure 1: MDS plots. Top row, without IDF. Bottom row, with IDF. Left column, un-normalized vectors. Middle column, normalized by word-count. Right column, normalized by Euclidean length. Red circles are art, blue squares music. Only with both IDF weights and Euclidean-length normalization do we get anything like a reasonable separate of the two categories though admittedly in some cases the view is obscured by a few outliers, which seem to compress the other points into a single blob. 9

10 4. Comment the sq.euc.dist function that is, go over it and explain, in English, what each line does, and how the lines work together to calculate the function. Answer: The key observation is that x y 2 = x 2 2 x y + y 2 for any vectors x and y. (Exercise: show this!) The function uses this to efficiently compute the squared distance, avoiding explicit iteration loops (which are slow in R) and avoid computing the same x 2 and y 2 terms over and over again. sq.euc.dist <- function(x,y=x) { # Set default value for second argument x <- as.matrix(x) # Convert argument to a matrix to make sure it can be # manipulated as such y <- as.matrix(y) # Ditto nr=nrow(x) # Count the number of rows of the first argument # This is the number of rows in the output matrix nc=nrow(y) # Count number of rows in the second argument # This is the number of COLUMNS in the output x2 <- rowsums(x^2) # Find sum of squares for each x vector xsq = matrix(x2,nrow=nr,ncol=nc) # Make a matrix where each COLUMN is a copy # of x2 --- see help(matrix) for details on # "recycling" of arguments --- matrix is # sized for output y2 <- rowsums(y^2) # Find sum of squares for each y vector # Make a matrix where each ROW is a copy of y2, again sized for output ysq = matrix(y2,nrow=nr,ncol=nc,byrow=true) # Make a matrix whose [i,j] entry is the dot product of x[i,] and y[j,] xy = x %*% t(y) # You should check that this has nr rows and nc columns! d = xsq + ysq - 2*xy # Add partial result matrices # Remember that for vectors x and y # x-y ^2 = x ^2-2x*y + y ^2 # writing * for the dot product. So d has as its [i,j] entry the squared # norm of x[i,] plus the squared norm of y[j,] minus twice their dot product # Make the diagonal EXACTLY zero if the two arguments are the same if(identical(x,y)) diag(d) = 0 # Need to use the identical() function to see if two whole objects are the # same --- using "x==y" here would give a MATRIX of Boolean values # Distances are >= 0, negative values are presumably from numerical errors in # calculating numbers close to zero; fix them d[which(d < 0)] = 0 return(d) # Return the tidied-up matrix of squared distances This is more detailed than it strictly needs to be. 10

11 5. (a) Explain what the cosine distance has to do with cosines. Answer: From vector algebra, we know that for any vectors x and y, x y x i y i = x y cos θ i where θ is the angle between the vectors. The cosine distance is the dot product divided by the product of the norms, so it s that cosine. (b) Calculate, by hand, the cosine distances between the three vectors in question 2. Answer: The cosine distance between the first and the third vector is clearly 1, and between either of them and the second vector is (c) Write a function to calculate the matrix of cosine distances (really, similarities) between all the vectors in a data-frame. Hint: you may want to use the distances function. Check that your function agrees with your answer to the previous part. Answer: The distances function can take an optional argument which is a function to apply to each pair of vectors from the two matrices. So first let s define functions which compute the dot product, Euclidean norm, and the cosine distance for a pair of vectors. dot.product = function(x,y) { return(sum(x*y)) # With vectors, * does component-by-component multiplication euc.norm = function(x) { return(sqrt(dot.product(x,x))) cosine.dist.pair = function(x,y) { return(dot.product(x,y)/(euc.norm(x)*euc.norm(y))) Now our real function is easy: cosine.dist.1 = function(x) { return(distances(x,fun=cosine.dist.pair)) This is not the slickest way to do it, because we calculate the norm of each vector 2n 1 times. (Exercise: Why is it 2n 1?) This is faster, at least when n is large: cosine.dist.2 = function(x) { y = div.by.euc.length(x) return(distances(y,fun=dot.product)) 11

12 Exercise: Why does this work? How does this avoid recomputing the norms of the vectors? Let s check this: > question2vectors <- matrix(c(1,0,0,1,4,5,10,0,0),nrow=3,byrow=true) > question2vectors [,1] [,2] [,3] [1,] [2,] [3,] > round(cosine.dist.1(question2vectors),2) [,1] [,2] [,3] [1,] [2,] [3,] > round(cosine.dist.2(question2vectors),2) [,1] [,2] [,3] [1,] [2,] [3,] so both versions work. 6. Write a function to find the document which best matches a given query string. The function should take two arguments, (1) The query, as a single character string, and (2) the bag-of-words matrix, and return the row number corresponding to the best-matching document. You can pick the distance measurement, but you should include inverse document-frequency weighting. Answer: The easiest way to do this is to use the nearest.points function in the provided code. The one tricky bit is making sure that the query string is turned into a bag-of-words vector using the same dictionary as the documents. The R manipulations here are based on the ones which standardize.ragged, in 01.R, uses to ensure that all the stories have a common lexicon. # Return the index of the document which best matches a given query string # First we convert the query string into a document vector, then into # a bag of words, and then we remove terms not in the lexicon of the data # frame storing the bag-of-words vectors for the documents # Note that these last removed words add the same amount to the (squared) # distance between the query and any document s bag of words, so those words # do not change which document is closest # Inputs: query, as a character vector (query) # data frame of bag-of-word vectors (BoW.frame) # Presumes: a data frame has been created for bag-of-word vectors, column # names being words 12

13 # Calls: strip.text(), get.idf.weight(), nearest.points() # Output: Index of the document, name of the corresponding row in data frame query.by.similarity <- function(query,bow.frame) { # Prepare the query BoW vector for comparison with the target documents query.vec = strip.text(query) # Turn the query into a vector of words query.bow = table(query.vec) # Turn it into a bag of words lexicon = colnames(bow.frame) # PRESUMES: BoW.frame has been made so words are # the column names query.vocab = names(query.bow) # What words appear in the query? # Restrict the query to words in the lexicon query.lex = query.bow[intersect(query.vocab,lexicon)] # Add zero entries for lexicon words not found in the query query.lex[setdiff(lexicon,query.vocab)] = 0 # query.lex now has all the right entries, but the words are out of order! query.lex = query.lex[lexicon] # Why does that work? # Finally, turn it into a one-row matrix q = t(as.matrix(query.lex)) # Do IDF scaling on the targets AND the query # The function idf.weight calculates the weights and re-scales the # data-frame but we need the weights to re-scale the query as well so we # write our own function get.idf.weights() --- see below # Get the weights idf = get.idf.weights(bow.frame) # Scale the columns in BoW.frame BoW = scale.cols(bow.frame,idf) # Scale the columns in q q = q * idf # Scale the rows by Euclidean length BoW = div.by.euc.length(bow) # Ditto the query q = q/sqrt(sum(q^2)) # Find the closest match between q and BoW.idf best.index = nearest.points(q,bow)$which # Use row names from the data for a little extra comprehensibility best.name = rownames(bow)[best.index] return(list(best.index=best.index,best.name=best.name)) # Return the vector of IDF weights from a data frame # The core of idf.weight(), but leaving out the actual re-scaling # Input: data frame # Output: weight vector get.idf.weights <- function(x) { 13

14 doc.freq <- colsums(x>0) doc.freq[doc.freq == 0] <- 1 w <- log(nrow(x)/doc.freq) return(w) Just having the row-number of the matching story is not quite so useful as having a more mnemonic label too, so I ve set the code up to also return row names. Let s make them: rownames(nyt.frame) = c(outer("art",1:57,paste,sep="."), outer("music",1:45,paste,sep=".")) We saw outer() already: the first instance here is going to create a vector reading art.1, art.2,...art.57, which will then be concatenated with music.1, music.2,...music.45, and this vector, of length 102, will become the row-names of the data frame. Testing: > query.by.similarity(art[[5]],nyt.frame) $best.index [1] 5 $best.name [1] "art.5" > query.by.similarity(art[[40]],nyt.frame) $best.index [1] 40 $best.name [1] "art.40" > query.by.similarity(music[[12]],nyt.frame) $best.index [1] 69 $best.name [1] "music.12" > query.by.similarity(music[[31]],nyt.frame) $best.index [1] 88 $best.name [1] "music.31" So it passes the basic check, and gives reasonable results for queries like abstract expressionism and rhythm and blues. 14