Arabic Text Classification Using N-Gram Frequency Statistics A Comparative Study

Transcription

1 Arabc Text Classfcaton Usng N-Gram Frequency Statstcs A Comparatve Study Lala Khresat Dept. of Computer Scence, Math and Physcs Farlegh Dcknson Unversty 285 Madson Ave, Madson NJ Khresat@fdu.edu Abstract- Ths paper presents the results of classfyng Arabc text documents usng the N-gram frequency statstcs technque employng a dssmlarty measure called the Manhattan dstance, and Dce s measure of smlarty. The Dce measure was used for comparson purposes. Results show that N-gram text classfcaton usng the Dce measure outperforms classfcaton usng the Manhattan measure. Keywords: N-gram, classfcaton, categorzaton, Arabc. I. INTRODUCTION The rapd growth of the Internet has ncreased the number of onlne documents avalable. Ths has led to the development of automated text and document classfcaton systems that are capable of automatcally organzng and classfyng documents. Text classfcaton (or categorzaton) s the process of structurng a set of documents accordng to a group structure that s known n advance. There are several dfferent methods for text classfcaton, ncludng statstcal-based algorthms, Bayesan classfcaton, dstance-based algorthms, k- nearest neghbors, decson tree-based methods [4] to name a few. Text classfcaton technques are used n many applcatons, ncludng e-mal flterng, mal routng, spam flterng, news montorng, sortng through dgtzed paper archves, automated ndexng of scentfc artcles, classfcaton of news stores and searchng for nterestng nformaton on the WWW. The maorty of these systems are desgned to handle documents wrtten n the Englsh language, and therefore are not applcable to documents wrtten n the Arabc language. Developng text classfcaton systems for Arabc documents s a challengng task due to the complex and rch nature of the Arabc language. The Arabc language conssts of 28 letters. The language s wrtten from rght to left. It has very complex morphology, and the maorty of words have a tr-letter root. The rest have ether a quadletter root, penta-letter root or hexa-letter root. Prevous work on Arabc text classfcaton has used dstance-based algorthms [5], Learnng algorthms [10], and Bayesan classfcaton methods [6] n developng automated text classfcaton systems. Specfcally, [8] used N-grams for searchng Arabc text documents. They nvestgated d-grams and tr-grams. No stemmng was performed. They concluded that the N-gram technque s not an effcent approach to corpus-based Arabc word conflaton. [9] used tr-grams for ndexng Arabc documents wthout any pror stemmng. The work of [11] uses N-grams wth and wthout stemmng for text searchng. Ther results ndcate that the use of tr-grams combned wth stemmng mproved the performance of search retreval, however, t was not statstcally sgnfcant. In ths paper the behavor of the N-Gram Frequency Statstcs technque for classfyng Arabc text documents s studed. The technque employs a dssmlarty measure called the Manhattan dstance, and Dce s measure of smlarty, for the purposes of classfcaton. The Dce measure was used for comparson purposes. Results show that N-gram text classfcaton usng the Dce measure gves better classfcaton results compared to the Manhattan measure. A corpus of Arabc text documents was collected from onlne Arabc newspapers. 40% of the corpus was used as tranng classes and the remanng 60% of the corpus was used for classfcaton. All documents, whether tranng documents or documents to be classfed went through a preprocessng phase removng punctuaton marks, stop words, dacrtcs, and non letters. For the tranng documents, the N-gram (N=3) frequency profle was generated for each document and saved n text fles. Then for each document to be classfed, the N-gram frequency profle was generated and compared aganst the N-gram frequency profles of all the tranng classes. The Manhattan and Dce measures were computed.

2 Usng the Manhattan measure, the category to whch a document belongs s the one wth the smallest Manhattan dstance, and usng the Dce measure, the category s the one wth the largest Dce measure. The classfcaton results usng these two measures were compared n terms of recall and precson. The rest of the paper s organzed as follows: n secton 2 the concept of N-grams s presented, secton 3 descrbes the text preprocessng phase and secton 4 gves detaled descrpton of the classfcaton procedure. Secton 5 presents the classfcaton results. II. N-GRAMS An N-gram [3] s an N-character slce of a strng. The N- gram method s language ndependent and works well n the case of nosy-text (text that contans typographcal errors). We used tr-grams for text classfcaton. The trgrams of a strng or token s a set of contnuous 3-letter slces of the strng. For example, the tr-grams for the word المودعين are: ا لم,لمو مود, ودع, دعي,عين. In general, a word of length w has w-2 tr-grams. Accordng to Zpf's law [12] : The nth most common word n a human language text occurs wth a frequency nversely proportonal to n Ths has the mplcaton that documents belongng to the same class or category wll have smlar N-gram frequency dstrbutons. Fgure 1 shows the Tr-gram frequency dstrbuton for a text document belongng to the sports category from our corpus. It clearly shows that the frequences of the most common Tr-grams are nversely proportonal to ther rank. Frequancy Dstrbuton for Tr-grams Frequency Tr-gram Rank Fgure 1. Frequency Dstrbuton of Tr-grams III. TEXT PREPROCESSING All text documents went through a preprocessng stage. Ths was necessary due to the varatons n the way text can be represented n Arabc. The preprocessng was performed for the documents to be classfed and the tranng classes themselves. Preprocessng conssted of the followng steps: 1) Convert text fles to UTF-8 encodng. 2) Remove punctuaton marks, dacrtcs, non letters, stop words. The defntons of these were obtaned from the Khoa stemmer [7].. ا wth ا إ,أ 3) Replace ntal.ئ wth ء followed by ى 4) Replace fnal

3 IV. N-GRAM BASED TEXT CLASSIFICATION A corpus of Arabc text documents was bult usng Arabc news artcles collected from onlne webstes of several Arabc newspapers. The corpus conssted of text documents coverng 4 categores: sports, economy, technology and weather. The technology and weather documents were very small n sze rangng from 1 KB to 4 KB. Sports and economy documents were much larger rangng from 2 KB to 15 KB for sports documents and 2 KB to 18 KB for economy documents. The smaller documents consttuted about 2% of the total number of documents n the sports and economy category. All these documents went through the text preprocessng step outlned above n secton 3. 40% of the corpus was selected as tranng classes, and the remanng 60% was used for testng the classfcaton procedure. The documents used for tranng went through the same procedures as dd the documents to be classfed. Specfcally, each document selected to be part of the tranng classes, was preprocessed as outlned above n secton 3. Then the N-gram profle was generated. Generatng the N-gram profle conssted of the followng steps: 1) Splt the text nto tokens consstng only of letters. All dgts are removed. 2) Compute all possble N-grams, for N=3 (Tr-grams) 3) Compute the frequency of occurrence of each N-gram. 4) Sort the N-grams accordng to ther frequences from most frequent to least frequent. Dscard the frequences 5) Ths gves us the N-gram profle for a document. For tranng class documents, the N-gram profles were saved n text fles. Each document to be classfed, went through the text preprocessng phase, then the N-gram profle was generated as descrbed above. The N-gram profle of each text document (document profle) was compared aganst the profles of all documents n the tranng classes (class profle) n terms of smlarty. Specfcally, two measures were used. The frst measure s a dstance or dssmlarty measure, called the Manhattan dstance [1]. It calculates a rank-order statstc for two profles by measurng the dfference n the postons of an N-gram n two dfferent profles. For each N-gram n the document profle, search for the N-gram n the class profle and calculate the dfference between ther postons. For N-grams not found n the class profle, a maxmum value s assgned. After all N-grams n the document profle have been exhausted, the sum of the dstance measures s computed. Manhattan ( P, P ) = k h = 1 ( P where P, P represent two N-gram profles The class that has the smallest Manhattan dstance s chosen as the class for the document beng classfed. The second measure used s the Dce measure [1] of 2 P P smlarty Dce( P, P ) = P + P Where P s the number of elements (N-grams) n profle P. Usng the Dce measure, the class wth the largest measure s chosen as the class for the text document beng classfed. The results obtaned usng the Manhattan measure and the Dce measure were compared n terms of precson and recall. Precson and recall are defned n [1] as follows: Where, CC Precson = TCF Recall CC = TC CC :number of correct categores(classes) found. TCF : total number of categores found TC : total number of correct categores h P h )

4 V. RESULTS To compare the performance of the tr-gram technque usng the Manhattan measure, and the Dce measure, the recall and precson values were computed. These values are shown n tables 1 and 2 respectvely. The best result for the tr-gram method usng the Manhattan measure was acheved for the sports category wth a recall value of 0.88, and the worst result was for the economy category wth a recall value of Table 1. Recall and precson usng Manhattan measure Statstcs for Manhattan measure Category Recall Precson Sports Economy Technology Weather category, followed by 0.98 for the sports category, and 0.89 for the economy category. The two measures produced equal low recall values for the technology category. The reason for ths s attrbuted to the nature of the newspaper artcles coverng technologcal ssues. They tend to be very dverse coverng a vast range of topcs. As a result, the tranng classes for the technology category dd not provde full coverage of all the dfferent topcs n the category. Overall, classfcaton usng the Dce measure outperformed classfcaton usng the Manhattan measure. The Manhattan measure has provded good classfcaton results for Englsh text documents [2]. The poor performance of the measure for Arabc n ths study, can be attrbuted to the nature of the Manhattan measure, and the complex morphologcal structure of Arabc, whch s qute dfferent than the structure for Englsh. Stemmng text documents before generatng the N-grams may gve us comparable results for the two measures. Table 2. Recall and precson usng Dce s measure Statstcs for Dce s measure category Recall Precson Sports Economy Technology Weather 1 1 The results for the tr-gram method usng the Dce measure exceed those for the Manhattan measure, reachng ts hghest recall value of 1 for the weather VI. CONCLUSION Ths paper presented the results of classfyng Arabc text documents usng the N-gram frequency statstcs technque employng a dssmlarty measure called the Manhattan dstance, and Dce s measure of smlarty. The Dce measure was used for comparson purposes. Results showed that N-gram text classfcaton usng the Dce measure outperforms classfcaton usng the Manhattan measure. REFERENCES [1] R. Baeza-Yates, and B. Rbero-Neto, Modern Informaton Retreval, Addson Wesley, [2] W. B. Cavnar, and J. M. Trenkle, N-Gram Based Text Categorzaton, Proceedngs of SDAIR-94, 3rd Annual Symposum on Document Analyss and Informaton Retreval, [3] M. Damashek, Gaugng Smlarty wth n-grams: Language-Independent Categorzaton of Text, Scence 267, pp , 10 February [4] M. H. Dunham, Data Mnng: Introductory and Advanced Topcs. Prentce Hall 2003 [5] R. M. Duwar, A Dstance-based Classfer for Arabc Text Categorzaton, In Proceedngs of the 2005 Internatonal Conference on Data Mnng, Las Vegas USA [6] M. El-Kourd, A. Bensad, and T. Rachd, Automatc Arabc document categorzaton based on the Naïve-Bayes Algorthm, Workshop on Computatonal Approaches to Arabc Scrpt-based Languages, COLING- 2004, Unversty of Geneva, Geneva, Swtzerland, August [7] S. Khoa, Personal communcaton. [8] H. S.Mustafa, and Q. Al-Radadeh Usng N-Grams for Arabc Text Searchng, Journal of the Amercan

5 Socety for Informaton Scence and Technology, 55(11), pp , [9] J. Savoy, and Y. Rasolofo, Report on the TREC-11 Experment: Arabc, Named Page and Topc Dstllaton Seraches, TREC [10] H. Sawaf, J. Zaplo, and H. Ney, Statstcal classfcaton methods for Arabc news artcles, Arabc Natural Language Processng n ACL2001, Toulouse France July [11] J. Xu, A. Fraser, and R. Weschedel, Emprcal Studes n Strateges for Arabc Retreval,. SIGIR 02 Tampere Fnland, [12] G. K. Zpf, Human Behavor and the Prncple of Least Effort, an Introducton to Human Ecology, Addson-Wesley, Readng, Mass., 1949.