CS646 (Fall 2016) Homework 1

Transcription

1 CS646 (Fall 2016) Homework 1 Deadline: 11:59pm, Sep 28th, 2016 (EST) Access the following resources before you start working on HW1: Download the corpus file on Moodle: acm corpus.gz (about 90 MB). Check out the starter code on GitHub: cs646_hw1. You can directly import the Maven project in IntelliJ. 1 Working with Corpus and Index (50 points) The XML-like corpus file acm corpus.gz includes the information of about 270,000 documents. After decompression, you can read and process it as a plain text file (using the UTF-8 character encoding). Each document contains the metadata and abstract of an academic article in the following format: <DOC> <DOCNO>ACM </DOCNO> <TEXT> Relevance based language models Victor Lavrenko, W. Bruce Croft Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval Abstract: We explore the relation between classical probabilistic models of information retrieval and the emerging language modeling approaches. It has long been recognized that the primary obstacle to effective performance of classical models is the need to 1

2 estimate a relevance model: probabilities of words in the relevant class. We propose a novel technique for estimating these probabilities using the query alone. We demonstrate that our technique can produce highly accurate relevance models, addressing important notions of synonymy and polysemy. Our experiments show relevance models outperforming baseline language modeling systems on TREC retrieval and TDT tracking tasks. The main contribution of this work is an effective formal method for estimating a relevance model with no training data. </TEXT> </DOC> The <DOCNO> field is a unique ID (docno) of the document. For example, the docno of the above document is ACM The <TEXT> field contains the title, the authors, the publication venue, and the abstract of the article (without any structure). This format is usually called the trectext format. Note that it is only an XML-like format. The corpus file is not a valid XML document. So you may not be able to parse the corpus file using regular XML APIs. Anyhow, it is very easy to parse a trectext format file from scratch. 1.1 Processing the Corpus (20 points) Write a program to iteratively read each document and its <DOCNO> and <TEXT> fields from the corpus file. You are free to use any programming languages for this part. You should tokenize the text of the <TEXT> field by any characters that are neither alphabetic characters (a-z and A-Z) nor digits (0-9). You should normalize the tokenized words into lowercase characters. You do not need to stem the words or remove stop words in HW1. For example, your program should tokenize Victor Lavrenko, W. Bruce Croft into the following five tokens: victor lavrenko w bruce croft Count and report the following statistics of the corpus: The total number of documents in the corpus (2 points). 2

3 The average length 1 of the <TEXT> field for a document (3 points). The number of unique words appeared in the whole corpus (excluding the <DOCNO> field) (3 points). Find the longest 2 document(s) (excluding the <DOCNO> field). Report the docno(s) of the longest document(s) and the length (3 points). The document frequency 3 of information and retrieval (2 points). Compute and report the IDF of the two words as well (2 points). IDF(w) = log N n w N: the total number of documents in the corpus. n w : the number of documents containing the word w. Find all documents containing both query and reformulation. Report the number of documents containing both words (2 points) and report the docnos of these documents (2 points). How long does it take for your program to count all these statistics? (1 points) Include these statistics in your report. Submit your source code for counting these statistics to Moodle along with your report. 1.2 Building an Index (10 points) Read the Lucene and Galago tutorials on GitHub 4. Build an index for the corpus acm corpus.gz using either Lucene or Galago (you can choose one by your preference). Do NOT apply any stemming or remove stop words when you build the index. Read the requirements of 1.3 in advance and make sure the index you created can provide an efficient way to compute the statistics in The length of a document field refers to the total number of words in that field. 2 by the total number of words 3 The number of documents containing the word

4 If you use Lucene, you can use LuceneBuildIndex.java as the starter code. Submit your building index program along with your report. If you use Galago, report your command for building index (including all the parameters) in the report. You do NOT need to submit the index folder to Moodle. 1.3 Accessing Index (20 points) Write a program to compute the following statistics based on the index you built in 1.2. Note that it is okay if the statistics based on the index have a slight difference with those you counted by your own in 1.1 (because the tokenization process is different). The total number of documents in the corpus (2 points). The average length of the <TEXT> field for a document (3 points). The number of unique words appeared in the whole corpus (excluding the <DOCNO> field) (3 points). Find the longest document(s) (excluding the <DOCNO> field). Report the docno(s) of the longest document(s) and the length (3 points). The document frequency of information and retrieval (2 points). Compute and report the IDF of the two words as well (2 points). Retrieve the posting lists for query and reformulation. Merge them using a Boolean AND operation. Report the number of documents in the merged list (2 points) and the docnos of each document (4 points). How long does it take for your program to count all these statistics based on the index? (1 points) Include these statistics in your report. Submit your source code for counting these statistics to Moodle along with your report. 4

5 2 Ranking Results (25 points) Use GalagoSearchIndex.java or LuceneSearchIndex.java as the starter code to implement the following three retrieval models. Your implementation should retrieve the posting lists of query terms from index, and then merge and sort the posting lists into a ranked list of search results. In HW1, you do NOT need to consider how to efficiently merge the posting lists. However, at least you should get your final results based on posting lists rather than a brute-force scan of all the indexed documents. You implementation should be robust enough to handle queries of arbitrary length. Do not change the output part of the starter code. Boolean AND (5 points): finding results containing all query terms. As we mentioned in class, Boolean search returns a set of results without defining how to rank them. However, in your program, please rank the retrieved set of results by their docnos in an ascending order (this helps us with grading). TF IDF (10 points): score(q, d) = freq(w, d) log N w q n w freq(w, d) stands for the frequency of the word w in the document d. n w is the document frequency of w in the whole corpus. N is the total number of documents in the corpus. VSM (cosine similarity) (10 points): score(q, d) = w q freq(w, q) freq(w, d) w q freq(w, q) 2 w d freq(w, d) 2 freq(w, q) and freq(w, d), respectively, stand for the frequency of the word w in the query q and the document d. We also use q and d for the set of unique words in the query and the document, respectively. 5

6 After you have implemented the models, simply run the starter code s main function. It will retrieve the results for the query query reformulation and output the search results of the three models to three different files (with the name results BooleanAND, results TFIDF, and results VSMCosine). It will also print out the top 10 results of the three models to the console. Submit your implementation and the three search result files to Moodle. You also need to include in your report the output of the console (the top 10 results information for each model). 3 Evaluation (25 points) We performed a small-scale relevance judgments for the search query query reformulation on the corpus. We stored the judgments as a file qrels. You can find the file in HW1 s Github repository. Each line of the file stores the docno of a search result and its relevance score by our assessment (1 indicates relevant and 0 stands for not relevant). Use Evaluation.java as the starter code and implement the following three effectiveness evaluation measures. Simply consider the unjudged results as not relevant (score 0). Precision at rank k (P@k): the precision of the top k search results. (5 points) P@k = r k k r k : the number of relevant results among the top k results. Recall at rank k (Recall@k): the recall of the top k search results. (5 points) Recall@k = r k R r k : the number of relevant results among the top k results. R : the total number of relevant resutls in qrels. Average Precision (AP). You should measure the average precision of the whole ranked list (counting all retrieved search results). (10 points) 6

7 AP = 1 R n relevance(i) P@i i=1 n: the total number of results in the ranked list. relevance(i): the relevance score (1 or 0) of the ith result. R : the total number of relevant resutls in qrels. After you have implemented the three effectiveness measures, you can run Evaluation.java s main function. It will print out some evaluation results for the three search result files to the console. Include the reported evaluation information to your report and discuss the effectiveness of the three retrieval models based on the evaluation results. (5 points) A A Checklist for Your Submission A report including the answers to all questions (in pdf). All your source codes. Write a brief readme file for the purpose of the program if necessary. The three search result files. Pack all the stuff as a.zip or.tar.gz file and upload to Moodle s HW1 submission link (before the deadline!). If you decide to use the 5-day extension (you can only use it once during the whole semester), send an and let the instructor and the TA know. 7