TEXT CATEGORIZATION PROBLEM

Transcription

1 TEXT CATEGORIZATION PROBLEM Emrah Cem Department of Electrical and Computer Engineering Koç University Istanbul, TURKEY Abstract Document categorization problem gained a lot of importance in the last years due to the increase in the number of digital documents. This paper analyzes the performance of different classification algorithms on text categorization problem. Importance of parameter optimization on the performance of the algorithms is also discussed. The paper mostly focuses on the SVM (Support Vector Machines) algorithm since, according to the literature, its performance is superior to other machine learning algorithms in text categorization problem. 1 Introduction Text categorization (TC) problem is to classify documents into a predefined number of categories. There are various application areas of TC such as document filtering, automatic meta-data generation, word sense disambiguation, and any document organization related applications [1]. This paper is organized as follows. In section 2, TC problem is defined formally, third section gives some statistical information about used dataset. Some ML algorithms are applied to TC problem in the third section. 2 Problem Definition TC task is actually a very generalized task. There are different types of TC task based on different constraints. For instance, single-label vs. multilabel TC.In single label TC task, each document is labeled with a unique symbol, whereas in multilabel TC, each document might be labeled with more than one label. My problem is actually a subcase of single label called binary TC because my documents are either in my special class(political or religious documents class) or not. Binary TC is actually more general than multilabel case, algorithms for binary TC may solve multilabel TC problems provided that classes are independent of each other. This is usually the case in many applications. For this reason, proposing algorithms with good performance for binary TC is crucial. Another type of TC task is category pivoted vs. document pivoted TC task. In Category pivoted TC, we ask the classes of a given document; whereas in document pivoted TC, we ask the documents that are in a given class. Hard vs. Rank TC is another category for TC. In a rank TC, given a document d i, we rank the classes based on the relatedness of each class to that document, instead of deciding to only one class. The opposite side is also possible. We may also rank the documents given a class c i.[1] E. Cem is with the Department of Computer Engineering, Koç University, Sariyer, Istanbul, TURKEY ecem@ku.edu.tr 1

2 3 Application Areas TC task has various application areas since the number of digital documents are increasing exponentially by the time. In order to control these enormous number of documents, people need some categorization algorithms. For instance, TC task is used for author identification, image categorization through analysis of textual captions [2], language identification of a word when its language is not known [3], and automated essay grading [4], etc. 4 Related Works There are various works on text categorization task. t is actually impossible to cite all TC related papers since it is more than 100. Some example studies [5], [6], [7] apply different ML algorithms and compare their results for TC problem. They show that SVM is more appropriate algorithm than other ML algorithms for TC problem. [1] has a very informative survey on automated text categorization even though it is published in When we consider more recent literature,[8] realizes a TC using n-gram model. They work on identifying authors of a document, gender of the author of a document, and classification of documents based on their genre. [9] proposes methods for how to find optimum SVM parameters for TC task. 5 Dataset Characteristics The data of NOVA come from the 20-Newsgroup dataset. This problem is a text-classifying problem. Each text to classify is an that was posted to one or several newsgroups. The aim is to separate politics and religion topics from all the other topics. The raw data comes as text files for the prior knowledge track. The topics are provided with the training data in that track. The preprocessed data for the agnostic track is a sparse binary representation using a bag-of-word with a vocabulary of approximately words. [10] 5.1 Number of examples and class distribution Table 1: General Statistics of NOVA Dataset Domain Sparsity Type FracPos Tr/Fn FeatNum Train Valid Test NOVA Text Mining %99.7 binary % Table 2: Prior knowledge data Positive Ex. Negative Ex. Total Training Set Validation Set Test Set Total All variables are binary. There are no missing values. The data is very sparse. Over 99% of the entries are zero. The data was saved as a sparse-binary matrix. 5.2 Benchmark Results If the most frequent class is always picked, accuracy will be (12547/17537)*100=% 72 6 ML Algorithms for Text Categorization Different ML algorithms is tried on NOVA dataset with different parameter values for each algorithm. 2

3 6.1 K-Nearest Neighbor Classifier In this classifier, while determining the class of a new unseen instance, only k nearest training set instances are considered. New data is put in a class which is the most common class among k nearest neighbors, by voting scheme. Figure 1: K-nearest neighbor classifier and PCA on TC task Since possibility of over-fitting data is high in K-NNC, by using PCA, most of the irrelevant or redundant features are eliminated (from to 100), which resulted in lower test errors. K-NNC is also very sensitive to scaling, hence normalization also reduced the test error significantly. 6.2 Naive Bayes Naive Bayes is a computationally inefficient algorithm. For this reason, firstly, PCA is applied, then input is normalized into [0-1]. Best result is taken when % 90 of variance is preserved and normalization is applied. This might be due to preventing over-fitting issue by dimensionality reduction and sensitivity to scaling problem by normalization. 6.3 Logistic Regression Logistic regression computes the linear classifier of a dataset by maximizing the likelihood criterion using the logistic (sigmoid) function. This algorithm becomes very slow for feature sizes. Hence, dimensionality reduction methods (PCA, KLM, and Fisher mapping)are used; however, the obtained classification error is always same, Neural Network Backward stepwise regression is applied this time. In backward stepwise regression begins with a full or saturated model and variables are eliminated from the model in an iterative process. The fit of the model is tested after the elimination of each variable to ensure that the model still adequately fits the data.when no more variables can be eliminated from the model, the analysis is completed. 3

4 Figure 2: Naive Bayes Errors on TC task Figure 3: Neural Network classifier Errors on TC task 4

5 Table 3: Errors for different kernel types Parameter Polynomial Homogeneous Exponential Radial Cosine SVM SVMs are actually set of supervised learning methods used for both classification and regression. There are various parameters that needs to be optimized in SVM. Hence, brute force method is not logical. There is a study that analyzes the affect of different parameters of text-classification problem [9]. They also evaluate the stability of TC performance. Figure 4: Neural Network classifier Errors on TC task I also tried 6 kernel types by keeping regularization parameter constant. I got the following errors: 7 Conclusion I have some problems with my results, I never get an error less than 0,2857 which seems to be a lower bound. However, it is actually a benchmark result. Since my paper aims to focus on SVM and show its superior performance on TC. It is not possible to explain my contributions to the literature, for now. References [1] Fabrizio Sebastiani. Machine learning in automated text categorization. ACM Comput. Surv., 34(1):1 47, [2] Carl L. Sable and Vasileios Hatzivassiloglou. Text-based approaches for non-topical image categorization. Int. J. on Digital Libraries, 3(3): ,

6 [3] William B. Cavnar. N-gram-based text filtering for trec-2. In TREC, pages , [4] Leah S. Larkey. Automatic essay grading using text categorization techniques. In SIGIR, pages 90 95, [5] Susan T. Dumais, John C. Platt, David Hecherman, and Mehran Sahami. Inductive learning algorithms and representations for text categorization. In CIKM, pages , [6] Edda Leopold and Jörg Kindermann. Text categorization with support vector machines. how to represent texts in input space? Machine Learning, 46(1-3): , [7] Yiming Yang and Xin Liu. A re-examination of text categorization methods. In SIGIR, pages 42 49, [8] M. Fatih Amasyali and Banu Diri. Automatic turkish text categorization in terms of author, genre and gender. In NLDB, pages , [9] Mikhail S. Ageev and Boris V. Dobrov. Support vector machine parameter optimization for text categorization problems. In ISTA, pages , [10] 6