NOTE: Your grade will be based on the correctness, efficiency and clarity.

Transcription

1 THE HONG KONG UNIVERSITY OF SCIENCE & TECHNOLOGY Department of Computer Science COMP328: Machine Learning Fall 2006 Assignment 1 Due Date and Time: October 20 (Fri), 2006, 12:00 noon. NOTE: Your grade will be based on the correctness, efficiency and clarity. Task: Digit Recognition In this assignment, you are required to implement the Naive Bayes Classifier for digit recognition on the USPS data set. Five-fold cross-validation will be used to select the best thresholding parameter. 1 Details 1.1 USPS Data Set The USPS data set contains grayscale handwritten digit images scanned from envelopes by the U.S. Postal Service (Figure 1). The images are of size 16 16, with pixel values in the range 0 to 2. The original training set contains 7,291 images, while the test set contains 2,007 images. In this assignment, we only use 1, 7, and 9 that are easily confused. We have extracted a training set ( train.txt ) and a test set ( test.txt ), of the format: line 1: [number of classes (integer)] [number of pixels (integer)] line 2...N + 1: (N is the number of images in the file) [class of pattern i (integer in the range 0 to number of classes-1)] [pixel value of pattern i (double)] line N + 2: 1 (an end marker) 1.2 Procedure 1. As the pixels are real-valued, we first convert these to 0 or 1 by using a threshold (thred). 1

2 Figure 1: Samples of the USPS digit images. If the pixel value is smaller than or equal to thred times the maximum pixel value, then set it to 0; otherwise, set it to 1. (Here, the maximum pixel value is the maximum over all the images in the training set). The threshold is chosen from the candidates: {0,0.05,0.1,0.15,0.2,0.25,0.3,...,0.85,0.9,0.95}, and is selected by 5-fold cross-validation using the naive Bayes classifier. For your convenience, we have already split the training data into 5 (equal-sized) folds and the files are fold1.txt, fold2.txt,..., fold5.txt. 2. Use the optimal threshold parameter obtained in Step 1 to convert the whole data set. 3. Train the naive Bayes classifier by using the whole training set. 4. Perform testing on the test data. To examine the usefulness of the cross-validation procedure in selecting the threshold parameter, you are required to draw the following two curves. 1. The first curve plots the average validation accuracy versus the threshold. 2. The second curve plots the testing accuracy versus the threshold. Note that in practice, you cannot use this method to select the threshold parameter as one should never look at the test set. 1 ). All the files related to this assignment can be downloaded from the tutorial homepage (link assignment 2

3 1.3 Command Syntax The program should be coded in C++ and run under UNIX. The source file must be named navbys.cpp, and the command syntax is: Here, navbys fold files test file output file fold files specifies the filenames containing the different folds of the training data for cross-validation. It is a string in the format foldi-j.txt where I and J are integers. For example, if you input fold1-5.txt, then the program will read 5 files named fold1.txt, fold2.txt,..., fold5.txt. test file is the file containing the test data. output file is the file for outputting the test results. Its format is as follows: line 1: [threshold (float)] line 2: [accuracy of the classifier (in %) (float)] line 3...N + 1: [predicted class labels of the test images (integer in the range 0 to number of classes-1)] line N + 3: 1 (an end marker) 1.4 Program Structure In the program, you should design the class NavBys and function cros val() as specified in the following: class NavBys { private: double** train data; int* train labl; double** val data; int* val labl; double** test data; int* test labl; 3

4 public: } double thrhd; double* prior; double** cndnl; double acc; char* output name; void get trainval(double** train set, int* train lbl, double** test set, int* test lbl); void get thrhd(double th); get test(double** test set, int* test lbl); void quntz trainval(); void quntz test(); void zero corr(); void training(); double validation(); void testing(); void output(); NavBys(); NavBys(); double cros val(string 5fold files, double* var set) { return optml thrhd; } The specifications of the variables and functions are: train data: 2-d array to store the training data, i.e., train data[i][j] is the grayscale value of the jth pixel in the ith training image; train labl: 1-d array whose ith entry is the label of the ith training image; val data: 2-d array to store the validation set; val labl: 1-d array to store the labels of the validation set; test data: 2-d array to store the test data, i.e., test data[i][j] is the grayscale value of the jth pixel in the ith test image; test labl: 1-d array whose ith entry is the true label of the ith test image; thrhd: threshold used to convert the grayscale pixel values to binary values. prior: 1-d array whose ith entry contains the prior probability of the ith digit class; 4

5 cndnl: 2-d array to store the conditional probabilities, i.e., cndnl[i][j] is the probability P(w j C i ) that the jth pixel is white given that the image comes from the ith class; acc: testing accuracy of the classifier; output name: name of the output file; get trainval(double** train set, int* train lbl, double** val set, int* val lbl): function to read the training and validation data. The four arguments should be passed on to class members train data, train labl, val data and val labl, respectively; get thrhd(double th): function to get the user-specified threshold, and then passed on to the class member thrhd; get test(double** test set, int* test lbl): function to read the test data. The two arguments should be passed on to the class members test data and test labl, respectively; quntz trainval(): function to threshold the training and validation data into binary values using the threshold thrhd; quntz test(): function to threshold the test data to binary values using the threshold thrhd; zero corr(): function to perform correction on zero class-conditional probabilities. You can either replace zero probabilities with a very small quantity or use the laplacian correction. training(): function to build the naive Bayes model. You should compute the prior (prior) and class conditional (cndnl) probabilities in this function; validation(): function to perform validation on the validation set, and returns the validation accuracy; testing(): function to perform classification on the test data. The accuracy is stored in acc; output(): function to output your results in the file output name; cros val(string 5fold files double* var set): function to perform 5-fold cross-validation. The 5fold files is a string whose value is passed on from the fold files in the command syntax. The var set contains the set of candidate threshold parameters. Note that inside this function: (1) you should properly combine 4 folds of data for training, and the remaining fold for validation; (2) the obtained training and validation data is then passed on to NavBys-typed object to perform a 5-fold cross validation. The function then returns the optimal threshold. Note: It is up to you to use 1-d (instead of 2-d) arrays to store the training, testing or validation data. For numerical stability you may use logarithm of the probabilities. 5

6 2 Submission We will collect your program using the CASS. Please submit the following files: 1. Well-documented program source code for navbys.cpp. 2. An output file ( output.txt ) containing your test results. 3. A REPORT file, with your name, student ID, address, and short descriptions of your programs. You should draw two curves as mentioned in section 1.2, compare them and have some discussions on the selection of the threshold parameter. If you have some ideas on how to extract better features (one feature per pixel, not necessarily binary), you can also discuss them here. For more information of the CS UNIX account, please read IMPORTANT NOTE: you should NOT modify any file submitted after the assignment collection deadline. 3 Grading Basic requirement includes program clarity, documentation, and consistency with the assignment specification. Your program will be compiled under UNIX and tested on the given test data set to verify the performance you reported. You discussions in the report is also taken into consideration. 3.1 Late Submission We accept late submissions, but with the following penalties: submit on or before October 21, 12:00 noon: deduct 30%; submit on or before October 22, 12:00 noon: deduct 60%; submit after October 22, 12:00 noon: deduct 100%. 6