Statistical Bayesian Learning for Automatic Arabic Text Categorization

Transcription

1 Jounal of Compute Scence 7 (): 39-45, 20 ISSN Scence Publcatons Statstcal Baesan Leanng fo Automatc Aabc Text Categozaton Bassam Al-Salem and Mohd. Juzaddn Ab Azz Depatment of Compute Scence, Facult of Infomaton Technolog, Unvest Kebangsaan Malasa, Bang, 43600, Selango, Malasa Abstact: Poblem statement: The apd nceasng of onlne Aabc documents necesstated applng Text Categozaton technques that ae commonl used fo Englsh language to categoze them automatcall. The complex mopholog of Aabc language and ts lage vocabula sze make applng these technques dectl dffcult and costl n tme and effot. Appoach: We have nvestgated Baesan leanng models n ode to enhance Aabc ATC. Thee classfes based on Baesan theoem had been mplemented whch ae Smple Naïve Baes (NB), Mult-vaant Benoull Naïve Baes (MBNB) and Multnomal Naïve Baes (MNB) models. TREC-2002 Lght Stemme was appled fo Aabc stemmng. Fo text epesentaton, Bag-Of-Wod and chaacte-level n-gam wth the length 3, 4 and 5 ae used. In ode to educe the dmensonalt of featue space, the followng featue selecton methods: Mutual Infomaton, Ch-Squae statstc, Odds Rato and GSS-coeffcent wee used. Concluson: MBNB classfe outpefomed both of NB and MNB classfes. BOW epesentaton leads to the best classfcaton pefomance; nevetheless, usng chaacte-level n-gam leads to satsfng esults fo Aabc ATC based on Baesan leanng. Moeove, the use of featue selecton methods damatcall nceases the categozaton pefomance. Kewods: Aabc Text Categozaton, Baesan Leanng, Featue Selecton, Automatc Text Categozaton, Multnomal Naïve Baes, Multvaate Benoull Naïve Baes, Odds Rato (OR), Infomaton Gan (IG), Featue Selecton (FS), Mutual Infomaton (MI). INTRODUCTION Automatc Text Categozaton (ATC) s the task of assgnng a gven document to ts pedefned catego automatcall. In ecent eas, usng the compute n ou lfe leads to ncease the numbe of electonc documents and dgtal nfomaton. As a esult, ATC has become one of the most poweful technques fo oganzng the data. Instead of usng the classcal models of text classfcaton that consst of a set of logcal ules defned manuall, Machne Leanng (ML) appoach had been appled wdel to classf the texts automatcall wth hgh accuac (Sebastan, 2002). The most common Supevsed ML algothms ae Statstcal Leanng algothms, whch povde a pobablt that a gven document beng assgned to patcula classes based on pobablstc model (Kotsants, 2007; Sebastan, 2002; Yang and Pedesen, 997). Baesan leanng model s statstcal leanng model that based on Baesan theoem of the ndependenc of featue tems gven the classfcaton. Naïve Baes (NB), Multvaate Benoull Naïve Baes (MBNB), Multnomal Naïve Baes (MNB) and Ngam, 998; Schnede, 2003; Mendez et al., 2008; Yang and Pedesen, 997) ae pobablstc models, whch all appl Baesan theoem whle the wa of computng the pobablt s dffeent.ml appoach dvded nto two phases; tanng phase and test phase. Fo the tanng phase, a set of documents of the collected copus (called tanng set) s used to buld the classfe b allotment a subset of the tanng set fo each catego and pocess them b seveal Infomaton Reteval (IR) technques to extact a set of featues used as chaactestcs fo each catego. In the test phase, the emande of the copus (called test set) wll be used to test and evaluate the pefomance of the classfe b classfng the documents unde each catego as unseen documents and then compae the estmated categoes to the pe-defned ones to measue the classfcaton pefomance. Tpcall, thee ae two epesentaton methods to epesent the text as a set of featues; Bag-Of- Wod (BOW) b usng sngle wods o phases as featues and n-gam b usng sequence of wods (Wod- Level n-gam) o chaactes (Chaacte Level n-gam) of the length n (El-Koud et al., 2004). One poblem ases of buldng ATC sstem s handlng the huge numbe of (Eheamend et al., 2003; Kotsants, 2007; McCallum Coespondng Autho: Bassam Al-Salem, Depatment of Compute Scence, Facult of Infomaton Technolog, Unvest Kebangsaan Malasa, Bang, 43600, Selango, Malasa 39

2 featues, whch can easl each odes of tens of thousands (Al-Hab et al., 2008; Eheamend et al., 2003). Fo educng the featue space dmenson, man IR technques have been appled, such as Stemmng, Stop-wods Removal and Featue Selecton (FS). FS technques such as Mutual Infomaton (MI), Ch-Squae Statstc (CHI), Infomaton Gan (IG), GSS Coeffcent (GSS) and Odds Rato (OR) used to educe the dmensonalt of featue space b elmnatng the featues that ae consdeed elevant fo a patcula catego (Al-Hab et al., 2008; Duw, 2007; Foman, 2003; Fagouds et al., 2005; Galavott et al., 2000). The man am of ths stud s to evaluate and enhance the pefomance of the followng Baesan-based classfes: NB, MBNB and MNB fo Aabc ATC and analss the effect of usng the followng FS methods: CHI, OR, MI and GSS on the classfcaton pefomance. Related wok: Man Baesan leanng and othe statstcal leanng models have been appled fo ATC. The bulk of ATC wok has been devoted fo Englsh and othe Latn language. Concenng Baesan leanng, McCallum and Ngam (McCallum and Ngam, 998) have caed out an analss stud of MNB and MBNB pefomance fo Englsh ACT. The esults poved that MNB outpefoms MNB. In addton, MNB can pefom well when the featue space sze deceased. Anothe stud conducted b Schnede (Schnede, 2003) of usng MBNB and MNB fo spam flteng. The fndngs confmed that MNB outpefoms MBNB. Unlke Englsh language, a lmted numbe of studes had been done fo Aabc ATC(Al-Hab et al., 2008; Dawsh and Oad, 2002; Duw, 2006; 2007; Haag et al., 2009; Kanaan et al., 2009; Khesat, 2009; Mesleh, 2008; 2007). Among all of them onl (Duw, 2006; 2007; Kanaan et al., 2009) used Baesan leanng model. Howeve, the emploed the smple NB, whle MNB and MBNB, whch we wll nvestgate n ths wok, ma acheve bette. Aabc language conssts of 28 lettes and unlke Englsh, t wtten fom ght to left.in addton, Aabc has a complex mopholog (El-Koud et al., 2004; Haat and Ass 2007). Fo that easons, applng ACT technques fo Aabc s moe complcated than that fo Englsh. METHODS AND MATERIALS Pelmna: Let X = {x : =,,n} be a fnte set of documents and let Y = { :j =,,m} be a fnte set of labels such that each document x ϵx belongs to a class label ϵy, gven a set of tanng examples, S = {(x, ), = (,,n)}. The Baesan leanng task s to buld fom the tanng set a pobablstc model capable of estmatng the condtonal pobablt of the class gven an example x,p( x), fo all possble values of and x. J. Compute Sc., 7 (): 39-45, Aabc Text Pe-pocessng: Lke n an ACT sstem, the fst step s pe-pocessng the plan texts. Fo Aabc texts, text pe-pocessng usuall nvolves the followng: emovng punctuaton maks, dactcs and non-aabc lettes, excludng the wods wth length less than thee and elmnatng stop-wods (Khesat, 2009; Lake et al., 2007). In ths stud, Aabc TREC-2002 Lght Stemme (Dawsh and Oad, 2002) s emploed to etun the wods to the stems b emovng the most fequent suffxes and pefxes. Featue Selecton: Gven a catego ϵy and a featue tem t belongs to one o moe documents n X. Let A denotes to the numbe of tmes t pesents n, B s the numbe of tmes t pesents wthout, C s the numbe of tmes t absents n, D s the numbe of tmes t absents wthout and n s the tanng set sze. CHI, MI, OR and GSS methods compute the scoe of t belongs to as the followng: ( ) CHI t, n ( AD CB) ( A + C)( B + C)( A + B)( C + D) n A () ( ) log 2 log 2 ( A + C)( A + B) (2) MI t, ( ) (3) OR t, AD CB AD - CB n ( ) (4) GSS t, 2 Max scoe of each FS functon calculated as Max = max FS(t, ) (5) scoe =,,m Max scoe etuns the appopate catego that t belong to. Classfes: Gven a document x epesented as a set of featue tems x = {t : =,.., x } and a catego. The condtonal pobablt of gven x,p( x), (called posteo pobablt) estmated as follows: ( ) ( ) ( ) p x = p t,,t = p x p(t ) (6) k = Thus, the Baes optmal classfe, the classfe that acheve the mnmum eo, s chosen accodng to: * = ag max p p(t ) = ( ) x (7)

3 J. Compute Sc., 7 (): 39-45, 20 Theefoe, the document x s classfed to the catego *. Patculal, f we denote to the numbe of documents unde whch contan t as n and the total numbe of documents unde as n. Then, the pobablt p(t ) s estmated usng Laplac po (Chen et al., 2009) as: + n p(t ) = m + n (8) whee, m s the numbe of all categoes and p() s the pobablt of computed as: Whle x does not depend on the catego, then, thee s no need to calculate p( x ) and x! Schnede (2004). Moeove, f we denote to the numbe of occuences of t n catego as n and the numbe of the tems n catego as n. So, the pobablt p(t ) s estmated b means of Laplac po as: p(t ) + n = (3) n + n whee, n s the total numbe of all documents and p() computed as: Numbe of documents n Total numbe of documents ( ) = (9) p Numbe of selected tems n p( ) = featue set sze (4) Baesan classfe ntoduced so fa s the smple fom of Naïve Baes, fo smplct we call t NB. MBNB: Suppose that, the featue set that extacted fom the tanng set s T = {t,,t k }. In MBNB, each document x s epesented as a bna vecto v =< v,,vk > n whch v = f t occus n the document x (at least once), o v = 0 othewse. Thus, each document x s seen as a esult of k Benoull tals, whee fo each tal we decde whethe o not t occus n x. Unde the naïve Baes assumpton that the pobablt of each wod occung n a document s ndependent of othe wods gven the class label, the pobablt p(x ) s computed as a smple poduct: k ( ) = ( ) = ( ).( ( )) = v v p x p v p t p t (0) Theefoe, the maxmum posteo classfe s constucted as: * = ag max p p v { ( ). ( )} () whee, p(t ) and p() come fom Equaton (8) and (9) espectvel. MNB: MNB epesents the document x = {t,,t x }, as a vecto v =< v,,v x >, whee v s the numbe of occuence of t n x. The pobablt p(x ) computed as the multnomal dstbuton: Pefomance measues: The effectveness of ACT sstem can be measued b sotng the categozaton esult nto the followng: Fo each catego, suppose that the classfe pedctons ae summed up as follows: Tue Postve (TP) efes to the set of documents that assgned coectl to, False Postve (FP) efes to the set of documents ncoectl assgned to, False Negatve (FN) efes to the set of documents ncoectl not assgned to and Tue Negatve (TN) efes to the set of documents coectl not assgned to. Pecson and ecall: Pecson (p) and Recall () of a catego defned as: TP p = TP + FN (5) TP = TP + FP (6) F-measue: F-measue s the most wdel measue used to measue the classfcaton pefomance and computed as the hamonc mean of p and taken the fom: 2p. F = p + (7) Macoaveaged-F (Maco-F): Maco-F computed as the athmetc aveage ove F-measue of all categoes: v ( ) x p t p( x ) = p( v ) = p( x ). x! = v! (2) 4 Maco - F = F m (8)

4 J. Compute Sc., 7 (): 39-45, 20 EXPERMENTS AND RESULTS The dataset used n ths stud s n-house collectons of Aabc news conssts of 3,72 documents and fll nto the followng categoes: Ats, Econom, Poltcs and Spot. Dataset dvded nto,732 documents fo tanng and,440 documents fo test. Table shows how the dataset dvded fo tanng and test pe catego. The fst step s pe-pocessng the plan texts. The pe-pocessng nvolves tokenzaton, nomalzaton, stop-wods emoval and stemmng. Fo text epesentaton, we used chaacte-level n-gam of length 3, 4 and 5 and stemmed-wods. Afte epesentng the text, we extact fou dffeent featues sets, one fo each epesentaton methods. Then, we emploed FS methods fo educng the featues dmenson. The FS methods emploed n ou stud ae the followng: CHI, MI, OR and GSS. Table 2 shows the mpact of usng FS methods fo educng the numbe of selected featues as stemmed-wods. Then, we bult and taned the followng Baesan leanng models: NB, MBNB and MNB. Fo thee featue epesentaton methods, fou featue selecton technques and thee classfes, the numbe of expements caed out s 36 dffeent expements n whch the numbe of expements fo each classfe s 2 expements. In each expement, we evaluated the pefomance of each classfe on the test set usng dffeent numbe of the top most fequent tems n each featue set. The gven numbes of the top selected featues ae 200, 400, 600, 800,,000 and,200 featues. Table : The categoes and the tanng and test set At Poltcs Economc Spot Tanng set Test set Table 2: Stemmed-wods featue set sze fo each catego befoe and afte applng FS methods. FS method At Poltcs Economc Spot Wthout CHI GSS MI OR Fg. : Maco-F esultsusng 3-gam epesentaton Results usng 3-gam epesentaton: Fg. shows the Maco-F esults usng 3-gam epesentaton. It s clea that MNB classfe acheved the best pefomance usng GSS method. The best Maco-F esult obtaned b MNB s (0.92) when the numbe of featue tems s,000 o,200. The second Maco-F esult s acheved also b MNB wth OR, whch s (0.907) occued when the numbe of selected featues s,200 featue. MBNB comes afte MNB n whch the best esult obtaned s (0.902) usng OR when the Fg. 2: Maco-F esultsusng 4-gam epesentaton numbe of top tems s,000 o,200. Howeve, unlke Results usng 4-gam epesentaton: Fg. 2 shows NB and MNB, MBNB pefoms well wth small that MBNB acheved the best pefomance oveall numbe of featues. The best Maco-F esults obtaned usng 4-gam epesentaton. The best Maco-F b NB s (0.897) usng GSS when the numbe of top acheved b MBNB s (0.927) when the numbe of tems s,000 o,200. Concenng FS methods, CHI featues s,200 selected b OR o GSS. MNB and NB leads to the wost pefomance oveall, whle GSS obtan the lowest pefomance when the numbe of appoxmatel leads to the best pefomance. featues less than,000, whle the acheve bette when 42

5 J. Compute Sc., 7 (): 39-45, 20 the numbe of featues ove than,000. The best esult obtaned b NB s (0.924) wth,200 featues selected b CHI. MBNB acheved bette than NB n aveage, whle the best Maco-F acheved s (0.924) when the numbe of featues s,200 selected b OR method. Results usng 5-gam epesentaton: Fom Fg. 3, t s clea that MBNB acheved the best pefomance ove all. Howeve, nceasng the 5-gam featues moe than 400 s not effectve n lage. NB and MNB ae pefomng well when the numbe of featues s moe than,000. The best Maco-F acheved b MBNB s (0.934) occued b usng,000 featues selected b OR method.mnb classfe acheved bette than NB when the numbe of featues less than,000 and afte that NB outpefoms MNB; howeve, nceasng the numbe of featues enhances the pefomance of both MNB and NB. The best Maco-F esults acheved b NB and MNB ae (0.929) and (0.89) obtaned when the numbe of featues ae,000 and,200 espectvel. Fg. 3: Maco-F esultsusng 5-gam epesentaton Fg. 4: Maco-F esultsusng stemmed-wods 43 Results usng BOW epesentaton (stemmedwods): Fg. 4 shows that MBNB classfe acheved well usng small numbe of top featues, whle MNB s pefomng bette when the numbe of featues nceased. Howeve, MBNB outpefoms both MNB and NB n geneal. The best Maco-F esult acheved b MBNB s (0.94) usng 400 featues, selected b CHI method. Howeve usng MI to select the featues fo MBNB leads to (0.933) Maco-F, when the numbe of top featues s 800. NB outpefoms MNB when the numbe of featues less than 800 featues wth the best Maco-F value (0.933) usng 800 featues, selected b MI. DISCUSSION MBNB outpefomed MNB and NB oveall. The eason behnd that s the numbe of extacted featues fom the dataset s not too lage. Moeove, the dataset s small and balanced. These fndngs ae dealng wth McCallum and Ngam (998) fndngs. In the stud that conducted on Baesan leanng fo ATC, the ponted out that MBNB almost acheved accuate pefomance wth small numbe of featues less than,000 featues and when the dataset s balanced. Howeve, n ou stud MBNB outpefoms MNB fo the same eason mentoned above. Futhemoe, MNB can outpefom both of NB and MBNB when the numbe of featues s extemel lage and when the featues occued man tmes n the tanng data. Fo nstance, usng 3-gam epesentaton leads to ncease the featues occuences n the tanng data and as a esult, MNB acheved the best pefomance. In addton, usng the stemmed-wods as featues leads to the best pefomance among all the used text epesentaton technques, nevetheless usng chaacte level n-gam fo Baesan leanng models leads to accepted esults; howeve, 3-gam epesentaton leads to the pooest pefomance. Table 3: The best choosng of FS methods that leads to the best pefomance Classfe Featue tpe Numbe of top featues Ove MBNB 3-gam GSS GSS MI 4-gam CHI CHI OR 5-gam GSS CHI OR BOW MI GSS OR NB 3-gam GSS GSS MI 4-gam GSS GSS CHI 5-gam GSS CHI GSS BOW GSS MI OR MNB 3-gam MI GSS MI 4-gam CHI GSS OR 5-gam GSS MI GSS BOW OR GSS GSS

6 CONCLUSION In ths stud, we nvestgated the use of Baesan leanng fo Aabc Text Categozaton. Two epesentaton tpe wee used fo epesentng the text and fou featue selecton methods wee nvestgated to educe the featue space dmensonalt. The expemental esults on a collecton of Aabc news poved that MBNB outpefoms both MNB and NB oveall and usng BOW epesentaton leads to the best pefomance. Futhemoe, ou fndngs vefed that usng n-gam s not lmted on the dstance-based classfcaton models. In ou expement, we have nvestgated chaacte level n-gam to epesent Aabc texts fo ATC based on Baesan leanng and t leads to accepted esults. In addton, we have analzed the elevance of choosng an appopate featue selecton method wth the sze and tpe of the featues and the effectveness on each classfe pefomance, (Table 3) sums up these fndngs.the bestmaco-fobtaned ove all s (0.94), acheved b MBNB when the numbe of featues s 400, epesented b BOW and selected b CHI featue selecton method. In the futue, we wll expand the numbe of Aabc categoes to cove the most common categoes and we wll nclude the othe Baesan leanng classfes that wee not mentoned n ths stud. REFERENCES Al-Hab, S., A. Almuhaeb, A. Al-Thubat, M.S. Khosheed and A. Al-Rajeh, Automatc Aabc Text Classfcaton. 9es jounées ntenatonales d analse statstque des données textuelles, JADT, 08: Chen, J., H. Huang, S. Tan and Y. Qu, Featue Selecton Fo Text Classfcaton Wth Naïve Baes. Expet sstems wth applcatons. Int. J., 36: DOI: 0.06/j.eswa Dawsh, K. and D.W. Oad, CLIR Expements at Maland fo TREC-2002: evdence combnaton fo Aabc-Englsh eteval. Poceedngs of the th Text Reteval Confeence. (TRC 02), Ctseeex Publshe, Pennslvana State, pp: Duw, R.M., Machne leanng fo Aabc text categozaton. J. Am. Soc. Infom. Sc. Technol., 57: DOI: 0.002/as Duw, R., Aabc Text categozaton. Int. Aab J. Infom. Technol., 4: J. Compute Sc., 7 (): 39-45, El-Koud, M., A. Bensad and T. Rachd, Automatc Aabc document categozaton based on the Naïve Baes Algothm. Poceedngs of the Wokshop on Computatonal Appoaches to Aabc Scpt-Based Languages, (CAASL 04), Stoudsbug, PA, USA, pp:5-58. Eheamend, S., D. Lews and D. Madgan, On the nave baes model fo text categozaton. Ctseeex. Foman, G., An extensve empcal stud of featue selecton metcs fo text classfcaton. J. Mach. Lean. Res. ACM, 3: DOI: 0.62/ Fagouds, D., D. Meetaks and S. Lkothanasss, Best Tems: An effcent featue-selecton algothm fo text categozaton. Knowl. Infom. Sst., 8: DOI: 0.007/s Galavott, L., F. Sebastan and M. Sm, Expements on the use of featue selecton and negatve evdence n automated text categozaton. Res. Adv. Technol. Dgtal Lbaes, 923: DOI: 0.007/ _6 Haat, R.A. and O.E.Ass, CASRA+: A Colloqual Aabc Speech Recognton Applcaton. Am. J. Appled Sc., 4: DOI: / Haag, F., E. El-Qawasmeh and P. Pchappan Impovng Aabc text categozaton usng decson tees. Poceedngs of the st Intenatonal Confeence Netwoked Dgtal Technologes, Jul, 28-3, IEEE Xploe Pess, Ostava, pp: 0-5. DOI: 0.09/NDT Kanaan, G., R. Al-Shalab, S. Ghwanmeh and H. Al- Maadeed, A compason of textclassfcaton technques appled to Aabc Text. J. Am. Soc. Infom. Sc. Technol., 60: DOI:0.002/ASI Khesat, L., A machne leanng appoach fo Aabc text classfcaton usng N-gam fequenc statstcs. J. Infom., 3: Kotsants, S.B., Supevsed machne leanng: A evew of classfcaton technques. Infomatca, 3: Lake, L., L. Ballesteos and M. Connell, Lght stemmng fo Aabc nfomaton eteval. Aabc Comput. Mophol., 38: DOI: 0.007/ _2 McCallum, A. and K. Ngam, 998. A Compason Of Event Models Fo Nave Baes Text Classfcaton. Poceedng of the AAAI-98 Wokshop on Leanng fo Text Categozaton. (WLTC 98), Publshe Ctesee, Pennslvana State, pp: 4-48.

7 J. Compute Sc., 7 (): 39-45, 20 Mendez, J.R., I. Cd, D. Glez-Peña, M. Rocha and F. Fdez-Rveola, A compaatve mpact stud of attbute selecton technques on naïve baes spam fltes. Lect. Notes Comp. Sc., 5077: , DOI: 0.007/ _7 Mesleh, A.W Suppot vecto machnes based Aabc language text classfcaton Sstem: Featue Selecton Compaatve Stud. Poceedngs of the 2th WSEAS Intenatonal Confeence on Appled Mathematcs, Dec. 29-3, Wold Scentfc and Engneeng Academ and Socet, Wsconsn, USA., pp: Mesleh, A.M.A., Ch squae featue extacton based SVMs Aabc language text categozaton sstem. J. Comput. Sc., 3: DOI:0.3844/JCSSP Sebastan, F., Machne leanng n automated text categozaton. ACM Comput. Suves (CSUR), 34: -47. DOI:0.45/ Schnede, K., A compason of event models fo nave baes ant-spam e-mal flteng. Poceedng of the 0th Confeence on Euopean Chapte of the Assocaton fo Computatonal Lngustcs, (ECACL 03) ACM, Budapest, Hunga, pp: DOI: 0.35/ Yang, Y. and J. Pedesen, 997. A Compaatve Stud on Featue Selecton n Text Categozaton. Poceedngs of the Fouteenth Intenatonal Confeence on Machne Leanng (ICML'97), Mogan Kaufmann Publshes Inc, CA, USA., pp: