A Comparative Impact Study of Attribute Selection Techniques on Naïve Bayes Spam Filters

Transcription

1 A Compaative Impact Study of Attibute Selection Techniques on Naïve Bayes Spam Filtes J. R. Méndez, I. Cid, D. Glez-Peña, F. Fdez-Riveola, M. Rocha 2 Dept. Infomática, Univesity of Vigo, Escuela Supeio de Ingenieía Infomática Edificio Politécnico, Campus Univesitaio As Lagoas s/n, 32004, Ouense, Spain {moncho.mendez icgomez dgpena iveola}@uvigo.es 2 Dept. Infomática, Univesity of Minho, Cento de Ciências e Tecnologias da Computação. Campus de Gualta, , Baga, Potugal mocha@di.uminho.pt Abstact. The main poblem of the Intenet sevice is the massive spam message delivey. Eveyday, hundeds of unwanted and unhelpful messages ae eceived by Intenet uses flooding thei mailboxes. Fotunately, nowadays thee ae diffeent kinds of filtes able to identify and automatically delete most of these messages. In ode to educe the poblem dimensionality only epesentative attibutes ae selected fom each using featue selection techniques. This wok pesents a compaison among five well-known featue selection stategies when they ae applied in conunction with fou diffeent types of Naïve Bayes classifies. The esults obtained fom the expeiments caied out show the elevance of choosing an appopiate featue selection technique in ode to obtain accuate esults. Intoduction and Motivation Duing the last yeas, Intenet has become an extemely impotant communication and infomation exchange platfom. In this context, elevant authoities have intoduced some stategic plans in ode to pomote the usage of Intenet and the newest communication technologies. Included into this plans, we can cite i200, a Euopean stategic pogamme designed to boost economical and social evolution based on a new society of knowledge []. Fo the development of these plans, seveal socioeconomic aspects, communication infastuctues, and education level fo the infomation society have been evaluated. Nevetheless, although the toubles caused by the delivey of spam messages ae upseting Intenet uses, these pogammes have not taken this fact into consideation. We believe that sevice and electonic delivey of instant messages ae one key matte fo the evolution of the infomation society. In this context, thee is a compelling need fo inceasing the classification accuacy of existing filtes in ode to automatically detect and dop spam messages. Nowadays, Intenet sevice is usually used by the vast maoity of the Intenet uses [2]. This fact ensues the efficacy of the advetising messages sent though Intenet [3]. Spam identification is a difficult task because spammes (spam message sendes) use ticks in ode to avoid spam filtes and ensue thei deliveies.

2 Moeove, the misclassification eos of anti-spam systems pesent diffeent cost values. False positive eos (legitimate messages classified as spam) ae unacceptable fo a geat amount of uses. Fom a complementay point of view, the pesence of a false negative eo (spam message classified as legitimate) is less hamful than a false positive misclassification. Geneally speaking, thee ae two kind of spam filteing techniques: (i) collaboative systems and (ii) content-based appoaches. The fome ae based on shaing identifying infomation fom spam messages in a filteing community while the late ae based on a deep analysis of the message content in ode to identify its class (usually using machine leaning techniques). This wok is focused in Naïve Bayes filtes, an accuate and well-known content-based technique based on combining the pobability of finding tems in spam and legitimate messages. Content-based spam filtes analyze the wods extacted fom the available messages. Although, each tem could be a candidate featue that should be analyzed fo spam filteing, this is not possible in pactice because the amount of wods extacted fom the whole copus is vey lage. The usage of lage featue vectos with machine leaning techniques is not advisable because it can cause the loss of efficiency and accuacy in existing filtes [4]. Theefoe, seveal featue eduction techniques need to be applied as a pe-pocessing stage pevious to the constuction of any spam filteing system. These techniques have been designed fo the dimensionality eduction and thei goal is to discad attibutes that do not povide essential infomation fo the classification task. This wok pesents a compaative study fo the impact of five featue selection methods when using fou vaiants of the oiginal Naïve Bayes algoithm woking as spam filte. The featue selection methods studied ae the following: (i) Infomation Gain (IG) (ii) Odds atio (OR), (iii) Document Fequency (DF) (iv) χ 2 statistic and (v) Mutual Infomation (MI). Moeove, we have analyzed the following Naïve Bayes altenatives: (i) Multivaiate Benoulli, (ii) Multinomial Naïve Bayes, (iii) Multivaiate Gaussian, and (iv) Flexible Bayes. The emaining of the pape is stuctued as follows: Section 2 pesents fou successful Naïve Bayes appoaches fo spam classification. Section 3 intoduces the details of seveal featue selection techniques used in the spam filteing domain. The expeimental potocol and the empiical evaluation esults ae showed in Section 4. Finally, Section 5 summaizes the main conclusions extacted fom this wok and outlines futue eseach lines. 2 Naïve Bayes fo Spam Filteing Nowadays, the vast maoity of commecial anti-spam filteing tools ae based on the usage of Naïve Bayes filtes [5, 6, 7]. This section pesents a compilation of diffeent ways of applying this technique fo spam filteing. Naïve Bayes classifies epesent each message, d, as a featue vecto in the fom x = { x,..., xm}, whee each x i stands fo the value of an attibute containing infomation about a token (tem o wod) identified in the taget . Keeping in mind

3 the Bayes theoem [2], the pobability of a given message belonging to the class c (spam, c s o legitimate, c l ) can be computed as shown in Expession (). p ( c) p( x c) pc ( x) = () px ( ) A Naïve Bayes classifie assigns to each message the class maximizing p c p x c whee p(c ) epesents the pobability of class c and p( x c ) is ( ) ( ) the pobability of finding a vecto x in the categoy c. Theefoe, in the spam filteing domain, we can classify a message d as spam when Expession (2) becomes tue: p(c s ) p( x c s ) p(c s ) p( x c s ) + p(c l ) p( x c l ) > T (2) whee c s and c l epesent the spam and legitimate classes espectively, and T is a theshold that epesents the equied secuity level included in the inteval [0, ]. The most common value fo T is 0.5 [8]. Thee ae seveal vaiants of the Naïve Bayes algoithm able to cope with the poblem of spam filteing. The main diffeence between them is the way to compute the pobability of finding a message that contains the featue vecto x consideing only those s belonging to class c, p( x c ). In this context, we can find the following altenatives: (i) Multivaiate Benoulli, (ii) Multinomial NB (iii) Multivaiate Gaussian and, (iv) Flexible Bayes. Multivaiate Benoulli is a vaiant of Naïve Bayes that epesents each message as a featue vecto x = { x,..., xm}, whee each element x i is a boolean attibute epesenting if the wod appeas on the taget message o not. Moeove, this vaiant consides that esults ae independent fo each categoy [9]. Multivaiate Benoulli computes the pobability p( x c ) as shown in Expession (3): ( ) ( x ) i p x c p t c p t c m xi ( ) = ( i ) ( i ) i= whee the pobability of finding a tem t i in a given message belonging to class c, p( t c ), is calculated using a Laplacean pio as shown in Expession (4). i (3) ( i c) p t + M = 2 + M ti, c c (4) whee M ti, cis the numbe of messages belonging to class c (spam o legitimate) that contains the tem t i and M c stands fo the amount of messages in class c.

4 Multinomial NB uses a vecto x = { x x },..., m in ode to epesent each message d, whee each element x i is epesented by a boolean attibute that indicates the pesence o absence of the tem t i in the message [2]. Moeove, thee is an altenative fo Multinomial NB based on epesenting each element x i as the numbe of appeaances of each tem t i in the message d. Nevetheless, this vaiant has showed to achieve p x c is computed as Expession (5) shows. poo esults [8]. In this case, ( ) i= ( i c) m p t p( x c ) = p( d ) d! x! whee x i is the value fo the tem t i and p(t i c ) is the pobability of finding documents containing the tem t i when only messages fom class c ae selected. As d is independent fom categoy c, thee is no need to calculate the sub-expessions p(d) and d! [0]. Moeove, each pobability p( ti c ) is estimated by means of a Laplacean pio as Expession (6) shows. + Nt i, c pt ( i c) = (6) m+ N whee N ti, c is the numbe of messages belonging to class c (spam o legitimate) that contain the tem t i, and N c stands fo the amount of messages belonging to class c. Multivaiate Gaussian uses continuous attibutes epesenting the fequency of the tems in a message. It assumes, in compaison with Multinomial NB, that the distibution associated to each tem is a Gaussian distibution fo each class c. Moeove, this vaiant consides that the values of the attibutes ae independent in each categoy [8]. The pobability p( x c ) is computed as Expession (7) shows c m px ( c) = gx ( ; μ ; σ ) i ic, ic, i= whee μic, and σ ic, epesent the mean and the standad deviation of the appeaance fequency of the tem t i in messages belonging to class c. Flexible Bayes [] woks in a simila way than Multivaiate Gaussian, but the distibutions of each attibute x i ae estimated by means of L i Gaussian distibutions epesenting each diffeent value fo the attibute in each class. Theefoe, the pob- p x c is computed as Expession (8) shows. ability ( ) i x i (5) (7)

5 m m Li ( ) = ( ) = i ( i; μlc,, σ, ) lc p x c p x c g x L i i i l= (8) whee L i epesents the numbe of diffeent values fo the attibute x i in the categoy c, μlc, is the l-th value fo attibute x i in categoy c and σ lc, epesents the standad deviation of the l-th value fo the attibute x i. In ode to simplify the calculation, σ is estimated as Expession (9) shows. lc, σ = lc, M c (9) whee M c is the numbe of messages belonging to categoy c []. Despite the simplicity of Naïve Bayes, the execution of featue eduction techniques is advisable in ode to dop unhelpful infomation gatheed fom s. Next section intoduces some featue selection techniques commonly used on spam filteing domain. 3 Featue Selection Techniques Thee ae a lot of available techniques fo featue selection puposes [2]. In this wok, we have consideed the usage of five well-known featue eduction stategies commonly used on the text mining domain. This section descibes the application of the following techniques: (i) Infomation Gain (IG), (ii) Odds atio (OR), (iii) Document Fequency (DF), (iv) χ 2 statistic and (v) Mutual Infomation (MI). IG is able to measue the numbe of bits that wee acquied fo the classification of spam and legitimate messages fom using the pesence o absence of a tem in a message [2]. If {c } n = epesents the set of categoies (spam o legitimate), the Infomation Gain of a tem t i is computed as Expession (0) shows. n ( i) = ( ) log ( ) IG t p c p c = n ( i) ( ) log ( i) + p t p c p c t = n ( i) ( ) log ( i) + p t p c p c t = whee p(t i ) is the pobability of finding a message with the tem t i, and p(c t i ) stands fo the pobability of an belonging to categoy c and containing the tem t i. OR is able to find tems commonly included in messages belonging to a cetain categoy. The meaning of this measue is the following: wods that appea in both (0)

6 spam and legitimate classes ae assigned an OR scoe nea to, othewise, tems with ae epesentative of a cetain class pesent an OR value highe than [3]. OR is computed as Expession () shows. ( i, ) OR t c ( i ) ( ) i p ( ti c) p( ti c) p t c p t c = i whee p( t c ) epesents the pobability of finding the tem t i in messages belonging to categoy c and ( i ) () p t c stands fo the pobability of finding the tem t i in e- mails fom opposite class to c. The DF measue epesents the numbe of messages whee a given tem appeas. Fo each tem t i, this method computes the numbe of s fom the taining copus containing it. Then those tems having lowe DF values ae discaded [2]. χ 2 statistic is able to test the hypothesis of the independency of two diffeent vaiables. Using a tem t i and a categoy c, Expession (2) shows the way to compute χ 2 statistic. χ 2 ( t, c ) = i 2 N ( A D C B) ( A+ C) ( B+ D) ( A+ B) ( C+ D) whee A is the amount of documents belonging to class c and containing the tem t i, B stands fo the numbe of documents containing the tem t i and belonging to the opposite categoy of c, C epesents the numbe of documents fom class c that do not contain the tem t i, D is the numbe of documents that does not contain the tem t i and does not belong to categoy c, and finally, N epesents the amount of available messages. χ 2 statistic eaches a value of 0 if t i and c ae independent. This method usually calculates the measue χ 2 fom each tem and categoy. The esults achieved fo the diffeent categoies can be computed as Expession (3) shows [2]. χ n ( t ) = p( c ) χ ( t, c ) 2 2 avg i i = (2) (3) MI is a technique fo statistic modelling of the language. Fo a given tem t i and a categoy c, MI scoe can be computed as Expession (4) shows. ( i ) p( ti c) A N ( ) p( c ) ( A+ C) ( A+ B) MI t, c = log log p t i whee A, B, C, D and N ae defined as peviously mentioned, p(t i ) epesents the pobability of finding a message with the tem t i, and p(t i c ) stands fo the pobability of finding a document fom categoy c containing the tem t i. Similaly to χ 2 statis- (4)

7 tic, MI(t i, c s ) and MI(t i, c l ) (The MI scoes fo the tem t i in spam and legitimate classes) can easily be combined as shown in Expession (3). Once the diffeent Naïve Bayes vaiants and the selected featue selection methods have been intoduced, next section pesents the expeimental potocol designed fo the empiical evaluation caied out as well as the esults achieved duing the execution of the expeiments. 4 Expeimental Setup and Results This section pesents the expeimental potocol designed fo the evaluation of the diffeent featue selection methods when they ae used in conunction with Naïve Bayes filtes. Moeove, we pesent the esults achieved duing the expeimentation caied out. Despite pivacy issues associated with legitimate messages, ecently some compilations have been published though Intenet. Fom these copuses, we highlight LingSpam [3] and SpamAssassin [4]. Given the geat amount of messages, we have selected the SpamAssassin copus fo the execution of the diffeent expeiments. Moeove, we have used only space chaactes as token sepaato and a stopwod emoval pe-pocessing technique as ecommended in [8]. All expeiments have been caied out using featue vectos containing 000 attibutes fo epesenting each message. Finally, in ode to ensue the quality of the achieved esults, we have used a 0 statified fold-coss validation [9]. Fo compaison puposes, we have used 5 well-known accuacy measues: (i) pecentage of false positives, false negatives and coect classifications, (ii) batting aveage, (iii) ecall and pecision, (iv) F-scoe and balanced F-scoe using diffeent β values and (v) Total Cost Ratio using diffeent values fo λ. Figue shows, the esults fo the false positive (FP), false negative (FN) and coect classifications (OK) achieved by the diffeent Naïve Bayes appoaches using the selected featue selection methods. The amount of FP eos should be caefully studied in ode to pevent the elimination of legitimate messages fom the inbox of final uses [5]. As we can see fom Figue (a), the wost method fo featue selection when using a Multivaiate Benoulli classifie is OR, while DF poduces the highest pecentage of coect classifications (90.67%) with a lowe amount of FP eos. Although, the MI method geneates the lowest FP eo ate, the numbe of eos achieved by using it is vey high. On the othe hand Figue (b), whee a Multinomial Naïve Bayes is used, OR pesents a poo pefomance (achieving the pecentages of 65% of FP eos and 9.43% of coect classifications). The method with the highest pecentage of coect classifications is DF (85.53%). Nevetheless, this method still pesents geat amount of FP eos (2%). Finally, although MI pesents the lowest FP ate, the pecentage of coect classifications is not vey good (72%). In Figue (c), the esults fo a Mutivaiate Gaussian NB ae shown and OR featue selection method behaves bette than othe techniques achieving a high pecentage of coect classifications (92.06%) as well as, the lowest amount of FP eos.

8 Finally, analysing the esults fom using a Flexible Bayes classifie in Figue (d), we can ealize that the OR featue selection technique achieves the highest amount of coect classifications (93.43%). Moeove, the use of this method does not intoduce FP eos. Theefoe, this combination epesents a good appoach fo spam filteing. IG also poduces a elatively high pefomance ate (75.08%) with a low amount of FP eos. Finally, the MI method should be discaded fo spam filteing woking with Flexible Bayes because it achieves a geat amount of FP eos (65%). 00% 00% 90% 90% 80% 80% 70% 70% 60% %FP 60% %FP 50% %FN 50% %FN 40% %OK 40% %OK 30% 30% 20% 20% 0% 0% 0% 0% MI CHI DF INFO OR (a) Multivaiate Benoulli MI CHI DF INFO OR (b) Multinomial NB 00% 00% 90% 90% 80% 80% 70% 70% 60% %FP 60% %FP 50% %FN 50% %FN 40% %OK 40% %OK 30% 30% 20% 20% 0% 0% 0% 0% MI CHI DF INFO OR (c) Multivaiate Gaussian MI CHI DF INFO OR (d) Flexible Bayes Figue. Pecentage of coect classifications, FP and FN eos ove the SpamAssassin copus Afte a pecentage accuacy evaluation we have analyzed pecision, ecall, F-scoe and balanced F-scoe measues fo the compaison of the selected poposals. Recall estimates the filte pefomance (a high scoe indicates that the vast maoity of spam messages ae detected) while pecision stands fo a secue opeation (geat scoe when no FP eos ae achieved). In ode to combine pecision and ecall measues two new measues have been ecently intoduced [6]. F-scoe is equal to when the evaluated classifie does not pesent eos. Balanced F-scoe woks similaly than F- scoe but it includes a weighted facto β in ode to establish the elevance between pecision and ecall. It can be computed as Expession (5) shows. f scoe β = (β 2 + ) pecision ecall β 2 pecision + ecall (5) If β is equal to, then F-scoe gets the same value than balanced F-scoe. Moeove, if β is geate than, pecision is consideed moe impotant that ecall. Othe-

9 wise, ecall is assumed as the most impotant citeion [6]. Table pesents a pecision and ecall analysis fo the use of diffeent featue selection methods in combination with Multivaiate Benoulli vaiant of Naïve Bayes. Table. Pecision, ecall, F-scoe and balanced F-scoe using Multivaiate Benoulli Pecision Recall F-scoe balanced F-scoe (β=2) A look at Table shows that in this case, DF clealy achieves the best scoes fo ecall and pecision. Table 2 shows a compaison of the analyzed featue selection techniques with Multinomial Naïve Bayes using the above mentioned measues. Table 2. Pecision, ecall, F-scoe and balanced F-scoe using Multinomial Naïve Bayes pecision ecall F-scoe balanced F-scoe (β=2) Analyzing Table 2, it is clea that the usage of OR in conunction with Multinomial NB geneates a poo pecision (geat amount of FP eos). Moeove, the DF method achieves the best value fo pecision. Finally, χ 2, DF and IG ae able to detect a geat amount of spam messages (ecall is geate). Table 3 pesents the pevious measues achieved by using the diffeent featue selection methods with Multivaiate Gaussian. Table 3. Pecision, ecall, F-scoe and balanced F-scoe using Multivaiate Gaussian pecision ecall 0, F-scoe 0, balanced F-scoe (β=2) 0, In this case, it is visible that the OR method pesents a high pecision ate as well as the highest ecall scoe. Finally, Table 4 shows the evaluation of the scenaio whee a Flexible Bayes classifie is used. In this scenaio, OR achieves the highest pecision level. This value means that thee ae no FP eos. Moeove, MI achieves a geat amount of FP eos. The analysis of F-scoe measue shows that OR is the most eliable featue selection method fo Flexible Bayes. Table 4. Pecision, ecall, F-scoe and balanced F-scoe β=2 using Flexible Bayes pecision ecall F-scoe balanced F-scoe (β=2)

10 We have also used batting aveage and TCR scoes fo compaison puposes. Batting aveage measues the ability to detect spam messages (effectiveness) and the capacity of geneating a small amount of FP eos (pecision). This measue is a pai hit ate/stike ate, whee the fome epesents the ate of spam messages detected and the late the amount FP eo ate [7]. TCR is a pefomance measue fom a cost point of view applied to spam filtes. It uses a λ paamete that shows the cost popotion between FP and FN eos. When TCR is computed, a FP is consideed λ times moe expensive than a FN eo [8]. It can be calculated as shown in Expession (6). TCR λ = nspam λ fp + fn whee fp and fn epesent the false positive and false negative amount and nspam stands fo the numbe of spam messages. High TCR scoes indicate low global cost while TCR scoes unde indicate that the usage of the classifie is wose than manually classifying the documents. Table 5 pesents the TCR and batting aveage esults achieved using the available featue eduction stategies with Multivaiate Benoulli. Table 5. TCR and batting aveage using Multivaiate Benoulli TCR λ= TCR λ= TCR λ= batting aveage 0.302/ 0.025/ / / /0.00 As we can see fom Table 5, DF, MI and χ 2 methods pesent the geatest pefomance in this scenaio. Moeove, the DF method obtains the lowest amount of FP eos. Table 6 shows the TCR and batting aveage scoes fo the diffeent featue selection methods when Multinomial Naïve Bayes filte is used. Table 6. TCR and batting aveage using Multinomial Naïve Bayes TCR λ= TCR λ= TCR λ= batting aveage 0.06/ / / / /0.346 As Table 6 shows, DF and χ 2 methods seem to be the most viable featue eduction stategies fo use in this scenaio. Table 7 shows the TCR and batting aveage scoes fo the diffeent featue selection methods using Multivaiate Gaussian. (6)

11 Table 7. TCR and batting aveage using Multivaiate Gaussian TCR λ= TCR λ= TCR λ= batting aveage 0.689/ / / / /0.76 As we can ealize fom Table 7, OR is clealy the best option fo use in this context. Table 8 shows the esults of batting aveage and TCR fo the diffeent featue selection stategies when using Flexible Bayes classifie. Table 8. TCR and batting aveage using Flexible Bayes TCR λ= TCR λ= TCR λ= Batting aveage 0.83/ / / / /0.329 Analyzing the TCR values showed in Table 8, we can see that the best method fo this scenaio is OR. Moeove, the est of the methods ae vey poo fom a cost pespective. Fom anothe point of view, OR pesents a stike ate equal to 0 guaanteeing the geat pefomance of Flexible Bayes woking with an OR featue selection. 5 Conclusions and Futhe Wok This pape pesents and compaes seveal featue selection methods commonly used in the context of text mining in conunction with fou diffeent Naïve Bayes classifies. In ode to evaluate the poposals, we have designed an empiical test using seveal measues commonly used in the domain of spam filteing. To sum up the main conclusions, the OR method pesents a high pefomance level when it is used with Gaussian-based Naïve Bayes algoithms. Nevetheless, the esults achieved by using Laplace-based Naïve Bayes vaiants ae vey poo. These appoaches wok bette when using DF, IG and χ 2 methods fo featue selection. Expeiments have shown that MI method should be discaded fo spam filteing. Fom anothe point of view, OR, χ 2, DF o IG methods should be selected taking into consideation the taget classifie. As futue wok, a pomising diection would be the possibility of using continuous updating stategies in conunction with Naïve Bayes spam filtes. The spam filteing domain is constantly changing and pesents the poblem of concept dift []. Theefoe, the usage of lazy leaning stategies should wok bette than the application of eage leaning algoithms. Moeove, some dynamical evaluation appoaches must be designed fo the filtes. This kind of assessment should pemit the evaluation of the impact of dynamical changes on the envionment.

12 Refeences. Euopean Commission: i200 - A Euopean Infomation Society fo gowth and employment. [ (2007) 2. CadCommunications - Siteband Copoation: Tends Repot [ (2007) 3. Cunningham, P., Nowlan, N., Delany, S. J., Haah, M.: A Case-Based Appoach to Spam Filteing than Can Tack Concept Dift. Poceedings of the 5th Intenational Confeence on Case Based Reasoning, ICCBR-2003, Wokshop of Long-Lived CBR Systems (2003), Jain, A., Zongke, D.: Featue Selection: Evaluation, Application, and Small Sample Pefomance, IEEE Tansactions on Patten Analysis and Machine Intelligence (997), Vol. 9(2), Petes, T., Robinson, G., Hooft, R., Hammond, M., Meye, T., Tue, S., Walke, A., Hindle, C., Pickett, N., Stone, T.: SpamBayes Poect. [ (2002) 6. Mozilla Poect: Mozilla Spam Filte. [ 7. Buton Compute Copoation: SpamPobe: A Fast Spam Bayesian Filte. [ (2002) 2. Domingos, P., Pazzani, M.: On the optimality of the simple Bayesian classifie unde zeoone loss. Machine Leaning (997), Vol. 29, Andoutsopoulos, I., Metsis, V., Paliouas, G.: Spam Filteing with Naive Bayes Which Naive Bayes?. Poceedings of the 3d Confeence on and AntiSpam (2006) 4. Andoutsopoulos, I., Koustias, J., Chandinos, K.V., Paliouas, G., Spyopoulos, C.: An Evaluation of Naïve Bayesian Anti-Spam Filteing. Poc. of the Wokshop on Machine Leaning in the New Infomation Age at th Euopean Confeence on Machine Leaning (2000), Schneide, K. M.: A compaison of event models fo Naive Bayes anti-spam filteing. 0th Confeence of the Euopean Chapte of the ACL, Budapest, Hungy (2003). John, G., Langley, P.: Estimating continuous distibutions in Bayesian classifies. Poceedings of the th Confeence on Uncetainty in Atificial Intelligence (995), Yang, Y., Pedesen, J.: A Compaative Study on Featue Selection in Text Categoization. Poceedings of the 4th Intenational Confeence on Machine Leaning (997), Andoutsopoulos, I.: Ling Spam Copus [ (2000) 4. Mason, J.: The Apache SpamAssassin Poect [ (2005) 5. Fdez-Riveola, F., Iglesias, E. L., Díaz, F., Méndez, J. R., Cochado, J. M.: Applying Lazy Leaning Algoithms to Tackle Concept Dift in Spam Filteing. Expet Systems With Applications, (2007) Vol. 33(), Shaw, W. M., Bugin, R., e Howell, P.: Pefomance standads and evaluations in IR test collections: Cluste-based etieval models. Infomation Pocessing and Management (997) 7. Gaham-Cumming, J.: Undestanding Spam Filte Accuacy. JGC spam and anti-spam newslette (2004) 8. Méndez, J. R., Iglesias, E. L., Fdez-Riveola, F., Díaz, F. y Cochado, J. M.: Tokenising, Stemming and Stopwod Removal on Anti-Spam Filteing Domain. Lectue notes in atificial intelligence (2006), Vol. 447, Kohavi, R.: A study of coss-validation and bootstap fo accuacy estimation and model selection. Poceedings of the 4th Intenational Joint Confeence on Atificial Intelligence (995), 37-43